-----============= acceptance-small: sanityn ============----- Fri Aug 21 22:25:30 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg326-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 22:25:52 (1787365552) === Starting client oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Mount client oleg326-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Started clients oleg326-client.virtnet: 192.168.203.126@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg326-client.virtnet: executing check_config_client /mnt/lustre oleg326-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg326-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9388c501a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9388c8476000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9388c501a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9388c8476000.idle_timeout=debug disable quota as required oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 22:26:34 (1787365594) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607664 s, 17.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7635] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 22:26:37 (1787365597) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 22:26:47 (1787365607) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 22:26:55 (1787365615) -rw-r--r-- 1 root root 0 Aug 21 22:26 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 22:27:03 (1787365623) -rw-r--r-- 1 root root 0 Aug 21 22:27 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (8s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 22:27:11 (1787365631) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 22:27:19 (1787365639) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 22:27:28 (1787365648) total 1 -rwxrwxrwx 1 root root 0 Aug 21 22:26 f2a -rwxrwxrwx 1 root root 0 Aug 21 22:27 f2b total 1 -rwxrwxrwx 1 root root 0 Aug 21 22:26 f2a -rwxrwxrwx 1 root root 0 Aug 21 22:27 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0113488 s, 45.1 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 22:27:35 (1787365655) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 22:27:39 (1787365659) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.484467 s, 4.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 22:27:47 (1787365667) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 22:27:54 (1787365674) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 22:28:02 (1787365682) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 22:28:09 (1787365689) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 22:28:15 (1787365695) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 22:28:22 (1787365702) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 22:28:29 (1787365709) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 22:28:38 (1787365718) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098643 s, 10.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0433614 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 22:28:46 (1787365726) 1+0 records in 1+0 records out 1 byte copied, 0.00394294 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0555578 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0250723 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0210476 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0184508 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0117656 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0093298 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0129167 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0090271 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0175057 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0396485 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00961033 s, 0.1 kB/s PASS 10a (10s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 22:28:56 (1787365736) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.034076 s, 90.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0200201 s, 205 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000912631 s, 3.4 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 22:29:03 (1787365743) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 22:29:10 (1787365750) warning: '-runtime' deprecated, use '-t runtime' instead - link 316 (time 1787365762.39 total 10.01 last 31.57) - link 609 (time 1787365772.40 total 20.01 last 29.28) - link 915 (time 1787365782.42 total 30.04 last 30.54) - link 1231 (time 1787365792.43 total 40.05 last 31.57) - link 1546 (time 1787365802.46 total 50.07 last 31.42) - link 1839 (time 1787365812.46 total 60.08 last 29.28) - link 2143 (time 1787365822.48 total 70.10 last 30.34) - link 2441 (time 1787365832.49 total 80.11 last 29.78) - link 2700 (time 1787365842.52 total 90.13 last 25.83) total: 2981 link in 100.01 seconds: 29.81 ops/second using seed 88879871 running for 100 seconds - stat 4645 (time 1787365764 ; total 11 ; last 11) - stat 9368 (time 1787365775 ; total 22 ; last 11) - stat 10000 (time 1787365776 ; total 23 ; last 1) - stat 14193 (time 1787365787 ; total 34 ; last 11) - stat 19195 (time 1787365798 ; total 45 ; last 11) - stat 20000 (time 1787365799 ; total 46 ; last 1) - stat 23206 (time 1787365810 ; total 57 ; last 11) - stat 26041 (time 1787365821 ; total 68 ; last 11) - stat 28368 (time 1787365832 ; total 79 ; last 11) - stat 30000 (time 1787365839 ; total 86 ; last 7) - stat 31856 (time 1787365850 ; total 97 ; last 11) total: 32597 stats in 100 seconds: 325.970001 stats/second - unlinked 0 (time 1787365803 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2981) error: No such file or directory total: 2980 unlinks in 89 seconds: 33.483147 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21437) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21439) - No such process PASS 12 (149s) == sanityn test 13: test directory page revocation ======= 22:31:38 (1787365898) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 22:31:48 (1787365908) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 22624 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 22:31:56 (1787365916) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 23203 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (9s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 22:32:04 (1787365924) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23782 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 22:32:11 (1787365931) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24365 Terminated $DIR1/$tdir/sleep 60 PASS 14c (9s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 22:32:20 (1787365940) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24948 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 22:32:31 (1787365951) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (56s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 22:33:27 (1787366007) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.97026 s, 8.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.508182 s, 16.5 MB/s Chance of close/open is 1 in 50 Seed set to 6011 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x581519 000005[0] 1787366012.817837 trunc from 00000000 to 0x581518 (0x581519 bytes) 000007[0] 1787366012.881697 write 0x748417 thru 0x751a12 (0x95fc bytes) truncating to largest ever: 0x8a3c87 000008[1] 1787366012.903056 trunc from 0x751a13 to 0x8a3c86 (0x152274 bytes) 000009[0] 1787366012.970373 mapread 0x151c37 thru 0x15839a (0x6764 bytes) 000010[0] 1787366013.031965 trunc from 0x8a3c87 to 0x2433d5 (0x6608b1 bytes) 000013[0] 1787366013.197840 mapread 0x14ead9 thru 0x1536c7 (0x4bef bytes) 000015[1] 1787366013.270714 mapwrite 0x28c7c4 thru 0x296e0f (0xa64c bytes) 000016[1] 1787366013.498024 write 0x80cf30 thru 0x816567 (0x9638 bytes) 000019[0] 1787366013.621637 trunc from 0x816568 to 0x46171e (0x3b4e49 bytes) 000021[0] 1787366013.723392 read 0x0cb1c3 thru 0x0d4994 (0x97d2 bytes) 000024[0] 1787366013.779880 mapread 0x0574f8 thru 0x05b4ae (0x3fb7 bytes) 000028[0] 1787366013.891073 trunc from 0x46171f to 0x848760 (0x3e7042 bytes) 000031[0] 1787366013.981983 mapwrite 0x576c86 thru 0x57afd4 (0x434f bytes) 000034[1] 1787366014.153885 read 0x2e8a35 thru 0x2e9de2 (0x13ae bytes) 000036[1] 1787366014.201008 read 0x24c193 thru 0x254a0c (0x887a bytes) 000037[0] 1787366014.210931 read 0x5ac421 thru 0x5af014 (0x2bf4 bytes) 000038[0] 1787366014.217689 read 0x66299c thru 0x6640a7 (0x170c bytes) 000041[1] 1787366014.240857 mapread 0x125f6b thru 0x12cb30 (0x6bc6 bytes) 000043[0] 1787366014.323631 mapwrite 0x134231 thru 0x13d030 (0x8e00 bytes) 000046[1] 1787366014.520288 mapwrite 0x462c60 thru 0x46755c (0x48fd bytes) 000047[0] 1787366014.805018 write 0x38d1cc thru 0x3919bb (0x47f0 bytes) 000048[0] 1787366014.854738 read 0x4fa3ce thru 0x4fae9d (0xad0 bytes) 000049[0] 1787366014.947704 mapread 0x113323 thru 0x117f7a (0x4c58 bytes) 000050[1] 1787366015.038831 trunc from 0x848761 to 0x3ea355 (0x45e40b bytes) 000054[1] 1787366015.230501 write 0x76b424 thru 0x776f60 (0xbb3d bytes) 000055[1] 1787366015.235607 mapwrite 0x7dda10 thru 0x7ec64d (0xec3e bytes) 000060[0] 1787366015.439859 mapwrite 0x278b67 thru 0x2820fa (0x9594 bytes) 000063[1] 1787366015.598319 read 0x40e440 thru 0x412577 (0x4138 bytes) 000064[0] 1787366015.628522 mapread 0x67c268 thru 0x685f77 (0x9d10 bytes) 000066[0] 1787366015.695092 mapread 0x3e1b43 thru 0x3e37ba (0x1c78 bytes) 000067[1] 1787366015.724580 trunc from 0x7ec64e to 0x869ae9 (0x7d49c bytes) 000076[1] 1787366016.004809 mapread 0x67f696 thru 0x68807e (0x89e9 bytes) truncating to largest ever: 0x8fa77a 000078[0] 1787366016.070988 trunc from 0x869aea to 0x8fa779 (0x90c90 bytes) 000081[1] 1787366016.144617 read 0x788264 thru 0x7906a4 (0x8441 bytes) 000084[0] 1787366016.221313 read 0x0bd09e thru 0x0c4eaa (0x7e0d bytes) 000086[1] 1787366016.247668 read 0x8081d2 thru 0x809efa (0x1d29 bytes) 000088[1] 1787366016.267593 write 0x948517 thru 0x94dd12 (0x57fc bytes) 000090[1] 1787366016.368639 read 0x045944 thru 0x04e446 (0x8b03 bytes) 000095[1] 1787366016.487018 mapwrite 0x53b743 thru 0x54637a (0xac38 bytes) 000096[0] 1787366016.604683 trunc from 0x94dd13 to 0x59d38f (0x3b0983 bytes) 000100[0] 1787366016.739482 read 0x1a790e thru 0x1aae32 (0x3525 bytes) 000104[1] 1787366016.773009 read 0x0ba596 thru 0x0c12b1 (0x6d1c bytes) 000106[1] 1787366016.823001 write 0x8fd025 thru 0x905822 (0x87fe bytes) 000108[1] 1787366016.867964 trunc from 0x905823 to 0x11bf1b (0x7e9907 bytes) 000109[0] 1787366016.921685 write 0x92d884 thru 0x930c5d (0x33da bytes) 000111[0] 1787366016.962723 read 0x2163b9 thru 0x219b53 (0x379b bytes) 000112[1] 1787366016.984753 write 0x7245f7 thru 0x72acf9 (0x6703 bytes) 000113[1] 1787366017.008451 write 0x89d78c thru 0x89e069 (0x8de bytes) 000119[1] 1787366017.114615 mapread 0x88ca01 thru 0x89c99d (0xff9d bytes) 000121[0] 1787366017.188500 write 0x47e904 thru 0x484cc5 (0x63c2 bytes) 000122[1] 1787366017.242199 mapread 0x192c61 thru 0x19315b (0x4fb bytes) 000123[1] 1787366017.278829 mapread 0x5bcf74 thru 0x5be74b (0x17d8 bytes) 000127[1] 1787366017.370552 mapwrite 0x560cdd thru 0x567ae7 (0x6e0b bytes) 000131[0] 1787366017.505050 mapwrite 0x63aa8d thru 0x63cc8e (0x2202 bytes) 000133[1] 1787366017.734766 read 0x523f04 thru 0x52bba3 (0x7ca0 bytes) 000135[1] 1787366017.788026 write 0x7731c4 thru 0x781fed (0xee2a bytes) 000137[1] 1787366017.827030 mapread 0x2d7635 thru 0x2e7613 (0xffdf bytes) 000138[0] 1787366017.901694 mapwrite 0x29a279 thru 0x2a3ae0 (0x9868 bytes) 000139[0] 1787366018.064861 trunc from 0x930c5e to 0x42e47c (0x5027e1 bytes) 000142[1] 1787366018.258839 mapread 0x1e335c thru 0x1eac1e (0x78c3 bytes) 000143[1] 1787366018.335886 write 0x986cc3 thru 0x9907e7 (0x9b25 bytes) 000144[1] 1787366018.396086 trunc from 0x9907e8 to 0x258b1a (0x737ccd bytes) 000146[0] 1787366018.496505 read 0x1513d0 thru 0x155c95 (0x48c6 bytes) 000149[0] 1787366018.623155 write 0x698f0f thru 0x6a241d (0x950f bytes) 000153[1] 1787366018.735404 write 0x91f6cd thru 0x9294a2 (0x9dd6 bytes) 000154[1] 1787366018.821910 read 0x5cc782 thru 0x5d06ba (0x3f39 bytes) 000156[0] 1787366018.949916 read 0x5f5aa7 thru 0x60457f (0xead9 bytes) 000158[1] 1787366019.016427 trunc from 0x9294a3 to 0x4ae248 (0x47b25a bytes) 000159[0] 1787366019.109910 mapread 0x390892 thru 0x393843 (0x2fb2 bytes) 000162[0] 1787366019.220765 mapwrite 0x2a827a thru 0x2b67fb (0xe582 bytes) 000163[1] 1787366019.411337 trunc from 0x4ae249 to 0x6c5390 (0x217148 bytes) 000168[1] 1787366019.501696 write 0x224aae thru 0x234673 (0xfbc6 bytes) 000169[1] 1787366019.525324 read 0x3b6f97 thru 0x3bf4bf (0x8529 bytes) 000170[1] 1787366019.538198 read 0x24b3d3 thru 0x24e24e (0x2e7c bytes) 000171[0] 1787366019.545222 mapread 0x19618f thru 0x1a36c2 (0xd534 bytes) 000172[0] 1787366019.676959 mapread 0x2a7a34 thru 0x2b4193 (0xc760 bytes) 000174[1] 1787366019.728895 read 0x45caf8 thru 0x468baf (0xc0b8 bytes) 000177[0] 1787366019.739028 mapread 0x42680e thru 0x42f540 (0x8d33 bytes) 000179[0] 1787366019.787019 mapread 0x30b10f thru 0x3181a7 (0xd099 bytes) 000180[1] 1787366019.813376 write 0x666fec thru 0x668a4e (0x1a63 bytes) 000184[0] 1787366019.858602 mapread 0x380138 thru 0x380fd9 (0xea2 bytes) 000194[0] 1787366020.000611 mapwrite 0x7c1024 thru 0x7c4904 (0x38e1 bytes) 000195[1] 1787366020.251565 read 0x19ac3c thru 0x1a2611 (0x79d6 bytes) 000197[1] 1787366020.307833 mapwrite 0x640bce thru 0x64c6a5 (0xbad8 bytes) 000199[1] 1787366020.590491 mapwrite 0x157b02 thru 0x1626ad (0xabac bytes) 000201[0] 1787366020.818979 trunc from 0x7c4905 to 0x1bcea7 (0x607a5d bytes) 000203[1] 1787366021.055858 trunc from 0x1bcea8 to 0x0c50ae (0xf7df9 bytes) 000204[0] 1787366021.204346 mapread 0x0842cb thru 0x092073 (0xdda9 bytes) 000207[0] 1787366021.414684 write 0x9b2b0e thru 0x9b3af7 (0xfea bytes) 000209[1] 1787366021.519339 mapread 0x65bf47 thru 0x662269 (0x6323 bytes) 000211[1] 1787366021.764613 mapwrite 0x033b07 thru 0x041cf3 (0xe1ed bytes) 000213[0] 1787366021.896923 read 0x2dd1a3 thru 0x2ddc74 (0xad2 bytes) 000214[0] 1787366021.958597 mapread 0x5eccdf thru 0x5fc4dc (0xf7fe bytes) 000215[0] 1787366022.003027 read 0x8a4399 thru 0x8b0991 (0xc5f9 bytes) 000216[1] 1787366022.019483 write 0x74a04e thru 0x7598c5 (0xf878 bytes) 000217[1] 1787366022.049283 read 0x061719 thru 0x07076f (0xf057 bytes) 000218[0] 1787366022.074852 mapwrite 0x537922 thru 0x53ddbb (0x649a bytes) 000219[0] 1787366022.177498 mapwrite 0x576c97 thru 0x581452 (0xa7bc bytes) 000221[1] 1787366022.270316 mapwrite 0x101cd1 thru 0x10301f (0x134f bytes) 000224[0] 1787366022.529833 mapwrite 0x75aa6c thru 0x760157 (0x56ec bytes) 000227[1] 1787366022.886668 mapwrite 0x65d10f thru 0x662565 (0x5457 bytes) 000229[0] 1787366023.063648 mapread 0x7d4d46 thru 0x7dd43c (0x86f7 bytes) 000230[1] 1787366023.256820 read 0x5fa6d3 thru 0x6045a8 (0x9ed6 bytes) 000237[1] 1787366023.339912 mapwrite 0x6eac1d thru 0x6ecf43 (0x2327 bytes) 000238[1] 1787366023.472883 read 0x792194 thru 0x79cf1a (0xad87 bytes) 000240[0] 1787366023.558679 mapwrite 0x76c1f8 thru 0x779bf2 (0xd9fb bytes) 000241[0] 1787366023.980276 mapread 0x7567a5 thru 0x760925 (0xa181 bytes) 000242[0] 1787366024.001542 trunc from 0x9b3af8 to 0x0b038c (0x90376b bytes) 000243[0] 1787366024.045660 mapread 0x06f3f6 thru 0x07d59e (0xe1a9 bytes) 000246[0] 1787366024.101079 mapwrite 0x5728e2 thru 0x5777ef (0x4f0e bytes) 000248[0] 1787366024.212008 trunc from 0x5777f0 to 0x642c31 (0xcb442 bytes) 000250[1] 1787366024.268884 mapwrite 0x5ac662 thru 0x5bc5b8 (0xff57 bytes) 000251[1] 1787366024.400782 write 0x3935bf thru 0x394073 (0xab5 bytes) 000254[1] 1787366024.427364 mapwrite 0x381327 thru 0x38982d (0x8507 bytes) 000255[1] 1787366024.652815 mapwrite 0x870279 thru 0x8798ac (0x9634 bytes) 000258[0] 1787366024.806836 mapread 0x38e878 thru 0x396914 (0x809d bytes) 000261[0] 1787366024.948341 mapwrite 0x4a7e7c thru 0x4acb57 (0x4cdc bytes) 000262[0] 1787366025.052594 mapread 0x7102ac thru 0x71e852 (0xe5a7 bytes) 000265[0] 1787366025.107748 mapread 0x0fad00 thru 0x101bd3 (0x6ed4 bytes) 000267[1] 1787366025.144733 write 0x9369d2 thru 0x93d56a (0x6b99 bytes) 000268[1] 1787366025.210003 trunc from 0x93d56b to 0x8d8251 (0x65319 bytes) 000269[0] 1787366025.272939 trunc from 0x8d8252 to 0x09e60c (0x839c45 bytes) 000271[1] 1787366025.369540 write 0x5528a8 thru 0x553af2 (0x124b bytes) 000272[1] 1787366025.395305 mapwrite 0x93324d thru 0x93f638 (0xc3ec bytes) 000273[1] 1787366025.488005 read 0x733e02 thru 0x73692a (0x2b29 bytes) 000274[0] 1787366025.494076 mapread 0x7a4af7 thru 0x7b42ed (0xf7f7 bytes) 000277[0] 1787366025.604962 mapread 0x0a9979 thru 0x0ae188 (0x4810 bytes) 000279[0] 1787366025.645598 mapread 0x430d45 thru 0x43b80f (0xaacb bytes) 000281[1] 1787366025.675986 trunc from 0x93f639 to 0x4ad580 (0x4920b8 bytes) 000282[0] 1787366025.723641 trunc from 0x4ad581 to 0x596d6e (0xe97ee bytes) 000284[0] 1787366025.767900 mapwrite 0x105b2d thru 0x10a51c (0x49f0 bytes) 000286[0] 1787366025.927069 read 0x455335 thru 0x4601ec (0xaeb8 bytes) 000287[1] 1787366025.941894 write 0x2e1666 thru 0x2e7be6 (0x6581 bytes) 000288[1] 1787366025.983284 write 0x4b7e0c thru 0x4b8461 (0x656 bytes) 000290[1] 1787366026.045857 write 0x0f4e4b thru 0x104d0d (0xfec3 bytes) 000292[1] 1787366026.115705 mapread 0x23e9cb thru 0x243006 (0x463c bytes) 000293[1] 1787366026.169791 mapread 0x31c009 thru 0x322889 (0x6881 bytes) 000295[0] 1787366026.226589 trunc from 0x596d6f to 0x73f6ee (0x1a8980 bytes) 000299[0] 1787366026.335799 mapwrite 0x38a759 thru 0x38f2ba (0x4b62 bytes) truncating to largest ever: 0x9cf483 000300[1] 1787366026.461791 trunc from 0x73f6ef to 0x9cf482 (0x28fd94 bytes) 000301[1] 1787366026.585340 mapread 0x2370bc thru 0x23b412 (0x4357 bytes) 000302[1] 1787366026.633607 mapwrite 0x3f6bec thru 0x403ab2 (0xcec7 bytes) 000303[0] 1787366026.858870 write 0x4c4c5c thru 0x4c7a87 (0x2e2c bytes) 000304[1] 1787366026.960275 read 0x3f0903 thru 0x3fb281 (0xa97f bytes) 000305[1] 1787366027.007736 trunc from 0x9cf483 to 0x64002a (0x38f458 bytes) 000309[1] 1787366027.243293 read 0x1bf199 thru 0x1c7cca (0x8b32 bytes) 000312[0] 1787366027.352690 read 0x4e3155 thru 0x4f04bc (0xd368 bytes) 000314[1] 1787366027.424571 trunc from 0x64002b to 0x9161e1 (0x2d61b7 bytes) 000316[0] 1787366027.516544 write 0x9dc406 thru 0x9dcb78 (0x773 bytes) 000317[0] 1787366027.549950 write 0x4829da thru 0x485b0b (0x3132 bytes) 000320[0] 1787366027.626071 mapwrite 0x377480 thru 0x383f7c (0xcafd bytes) 000322[0] 1787366027.733065 read 0x0b704c thru 0x0b756d (0x522 bytes) 000323[1] 1787366027.768493 read 0x28f636 thru 0x29d1fd (0xdbc8 bytes) 000325[0] 1787366027.810609 mapwrite 0x76e87b thru 0x770585 (0x1d0b bytes) 000328[0] 1787366027.962968 trunc from 0x9dcb79 to 0x5b063c (0x42c53c bytes) 000329[1] 1787366028.078872 read 0x5aeff7 thru 0x5b063c (0x1646 bytes) 000332[1] 1787366028.221496 trunc from 0x5b063d to 0x9a4df3 (0x3f47b7 bytes) 000334[0] 1787366028.318724 write 0x9d2203 thru 0x9dd811 (0xb60f bytes) 000335[0] 1787366028.359883 write 0x893513 thru 0x89528c (0x1d7a bytes) 000337[1] 1787366028.407748 mapwrite 0x7b8b5f thru 0x7c44d7 (0xb979 bytes) 000338[0] 1787366028.570531 mapwrite 0x5add8d thru 0x5af967 (0x1bdb bytes) 000341[0] 1787366028.785811 write 0x1e55f7 thru 0x1f1123 (0xbb2d bytes) 000342[1] 1787366028.811268 read 0x8c361e thru 0x8cc6a8 (0x908b bytes) 000344[0] 1787366028.876858 read 0x7f7819 thru 0x7fa6da (0x2ec2 bytes) 000347[0] 1787366028.940291 mapread 0x935fcb thru 0x93fbd4 (0x9c0a bytes) 000348[0] 1787366028.972427 write 0x96e063 thru 0x9736de (0x567c bytes) 000349[1] 1787366029.018958 write 0x8a6f58 thru 0x8accea (0x5d93 bytes) 000350[1] 1787366029.067728 mapread 0x991c79 thru 0x992826 (0xbae bytes) 000351[0] 1787366029.105096 trunc from 0x9dd812 to 0x0db789 (0x902088 bytes) 000353[1] 1787366029.196772 read 0x092178 thru 0x0927cb (0x654 bytes) 000356[1] 1787366029.345648 trunc from 0x0db78a to 0x24b368 (0x16fbdf bytes) 000357[1] 1787366029.427008 trunc from 0x24b369 to 0x4d9596 (0x28e22e bytes) 000360[0] 1787366029.485933 mapwrite 0x09d0a6 thru 0x0a1c2c (0x4b87 bytes) 000362[0] 1787366029.637989 read 0x1a1597 thru 0x1ad207 (0xbc71 bytes) 000364[0] 1787366029.705643 write 0x915183 thru 0x923896 (0xe714 bytes) 000367[1] 1787366029.815178 read 0x510863 thru 0x51fc30 (0xf3ce bytes) 000372[0] 1787366030.235561 mapread 0x43e90f thru 0x446c92 (0x8384 bytes) 000375[0] 1787366030.377506 mapwrite 0x12c36f thru 0x13807b (0xbd0d bytes) 000378[0] 1787366030.550613 mapwrite 0x3edfb9 thru 0x3f9a56 (0xba9e bytes) 000379[0] 1787366030.719308 read 0x51cf01 thru 0x5245fd (0x76fd bytes) 000383[0] 1787366030.910921 trunc from 0x923897 to 0x588d9a (0x39aafc bytes) 000384[0] 1787366030.993092 write 0x92df9a thru 0x93b317 (0xd37e bytes) 000388[1] 1787366031.061812 mapread 0x81b20c thru 0x82ab61 (0xf956 bytes) 000391[0] 1787366031.258735 write 0x747570 thru 0x7513b9 (0x9e4a bytes) 000393[0] 1787366031.278770 trunc from 0x93b318 to 0x379acd (0x5c184a bytes) 000396[1] 1787366031.417898 trunc from 0x379ace to 0x9cda52 (0x653f85 bytes) 000398[1] 1787366031.504368 write 0x89b9b3 thru 0x8a8059 (0xc6a7 bytes) 000399[0] 1787366031.540895 mapread 0x8f5ca8 thru 0x8f879a (0x2af3 bytes) 000403[0] 1787366031.662521 mapwrite 0x0873c0 thru 0x0939f9 (0xc63a bytes) 000405[1] 1787366031.842648 read 0x41cc5a thru 0x424a49 (0x7df0 bytes) 000407[1] 1787366031.899166 trunc from 0x9cda53 to 0x8cd57b (0x1004d7 bytes) 000413[0] 1787366032.001044 mapwrite 0x18b0cf thru 0x18dc72 (0x2ba4 bytes) 000414[1] 1787366032.111520 read 0x480902 thru 0x48d4fa (0xcbf9 bytes) 000415[0] 1787366032.142377 write 0x42c957 thru 0x4319f5 (0x509f bytes) 000418[1] 1787366032.229608 mapread 0x53e904 thru 0x547e6e (0x956b bytes) 000426[0] 1787366032.382785 trunc from 0x8cd57c to 0x627da3 (0x2a57d8 bytes) 000435[0] 1787366032.501539 write 0x4e96ee thru 0x4f2efc (0x980f bytes) 000438[0] 1787366032.567172 mapwrite 0x2c884e thru 0x2c899b (0x14e bytes) 000439[1] 1787366032.680949 mapread 0x17bb41 thru 0x18542e (0x98ee bytes) 000443[0] 1787366032.773830 trunc from 0x627da4 to 0x8ab988 (0x283be5 bytes) 000450[1] 1787366032.886476 write 0x009678 thru 0x0096aa (0x33 bytes) 000451[0] 1787366032.930984 trunc from 0x8ab989 to 0x37908f (0x5328f9 bytes) 000453[0] 1787366032.970876 mapwrite 0x0ad63d thru 0x0b3922 (0x62e6 bytes) 000454[0] 1787366033.099903 trunc from 0x379090 to 0x988299 (0x60f20a bytes) 000460[0] 1787366033.210275 mapwrite 0x425508 thru 0x4338d2 (0xe3cb bytes) 000461[0] 1787366033.359480 write 0x1ffc9f thru 0x209c2a (0x9f8c bytes) 000463[0] 1787366033.423656 mapread 0x303cc6 thru 0x313b49 (0xfe84 bytes) 000465[0] 1787366033.473383 read 0x00b6ca thru 0x0117d6 (0x610d bytes) 000468[0] 1787366033.542633 mapread 0x30388b thru 0x311feb (0xe761 bytes) 000470[1] 1787366033.653726 mapread 0x7069fe thru 0x709788 (0x2d8b bytes) 000472[1] 1787366033.755556 trunc from 0x98829a to 0x457e78 (0x530421 bytes) 000474[0] 1787366033.875918 write 0x981394 thru 0x98dbb0 (0xc81d bytes) 000483[1] 1787366034.158519 trunc from 0x98dbb1 to 0x082afd (0x90b0b3 bytes) 000484[1] 1787366034.226551 trunc from 0x082afe to 0x41fe5c (0x39d35f bytes) 000486[1] 1787366034.286390 write 0x6bab82 thru 0x6c3968 (0x8de7 bytes) 000487[0] 1787366034.288307 mapread 0x18b900 thru 0x18c202 (0x903 bytes) 000488[1] 1787366034.318064 trunc from 0x6c3969 to 0x3d8aac (0x2eaebc bytes) 000489[1] 1787366034.390712 mapread 0x15c5fb thru 0x16bc7c (0xf682 bytes) 000490[0] 1787366034.515200 mapread 0x23522f thru 0x23fc6c (0xaa3e bytes) 000495[1] 1787366034.667069 trunc from 0x3d8aad to 0x897366 (0x4be8ba bytes) 000496[1] 1787366034.724958 mapwrite 0x8cf5bd thru 0x8dd883 (0xe2c7 bytes) 000497[0] 1787366034.876972 write 0x694c54 thru 0x69e9ef (0x9d9c bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 6035 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[0] 1787366035.991876 write 0x359000 thru 0x35aa65 (0x1a66 bytes) 000004[0] 1787366036.003357 mapread 0x090000 thru 0x09b851 (0xb852 bytes) 000005[1] 1787366036.023792 read_OD 0x0a7000 thru 0x0affff (0x9000 bytes) 000008[0] 1787366036.053975 mapwrite 0x33a000 thru 0x33fec1 (0x5ec2 bytes) 000009[1] 1787366036.171009 mapwrite 0x590000 thru 0x597c38 (0x7c39 bytes) 000010[1] 1787366036.284686 mapwrite 0x30a000 thru 0x316457 (0xc458 bytes) 000011[0] 1787366036.462872 mapread 0x440000 thru 0x440e09 (0xe0a bytes) 000012[1] 1787366036.490712 write_OD 0x39f000 thru 0x39ffff (0x1000 bytes) 000016[1] 1787366036.593887 read_OD 0x03f000 thru 0x047fff (0x9000 bytes) 000020[0] 1787366036.615674 mapread 0x440000 thru 0x445779 (0x577a bytes) 000021[0] 1787366036.659596 mapwrite 0x336000 thru 0x33a119 (0x411a bytes) 000024[1] 1787366036.894463 mapwrite 0x37f000 thru 0x38514e (0x614f bytes) 000025[1] 1787366037.241522 read_OD 0x47b000 thru 0x481fff (0x7000 bytes) truncating to largest ever: 0x90d626 000026[1] 1787366037.270146 trunc from 0x597c39 to 0x90d625 (0x3759ed bytes) 000028[0] 1787366037.335805 trunc from 0x90d626 to 0x3aa03f (0x5635e6 bytes) 000029[1] 1787366037.374274 read_OD 0x25b000 thru 0x263fff (0x9000 bytes) 000030[0] 1787366037.415457 trunc from 0x3aa040 to 0x4af4f2 (0x1054b3 bytes) 000032[1] 1787366037.505578 read_OD 0x10e000 thru 0x114fff (0x7000 bytes) 000033[1] 1787366037.561262 read_OD 0x150000 thru 0x15dfff (0xe000 bytes) 000034[1] 1787366037.582231 mapwrite 0x834000 thru 0x83573f (0x1740 bytes) 000038[0] 1787366037.775760 mapread 0x1c8000 thru 0x1cf37a (0x737b bytes) 000039[0] 1787366037.811456 trunc from 0x835740 to 0x020fec (0x814753 bytes) 000040[0] 1787366037.858587 mapread 0x014000 thru 0x017cc9 (0x3cca bytes) 000042[1] 1787366037.883029 mapread 0x002000 thru 0x00f446 (0xd447 bytes) 000043[1] 1787366037.924688 mapread 0x002000 thru 0x0049c2 (0x29c3 bytes) 000044[0] 1787366037.925321 trunc from 0x020fed to 0x853c08 (0x832c1c bytes) 000046[1] 1787366038.029974 mapread 0x7a9000 thru 0x7b2a70 (0x9a71 bytes) 000048[1] 1787366038.079801 mapwrite 0x94e000 thru 0x958b0d (0xab0e bytes) 000049[1] 1787366038.189700 write_OD 0x2ec000 thru 0x2effff (0x4000 bytes) 000053[0] 1787366038.244646 trunc from 0x958b0e to 0x00ee96 (0x949c77 bytes) 000056[0] 1787366038.280139 write 0x4d1000 thru 0x4db5f7 (0xa5f8 bytes) 000057[1] 1787366038.323887 mapwrite 0x8de000 thru 0x8e9612 (0xb613 bytes) 000058[0] 1787366038.531332 trunc from 0x8e9613 to 0x01c5a3 (0x8cd06f bytes) 000060[1] 1787366038.622984 write_OD 0x5ae000 thru 0x5affff (0x2000 bytes) 000062[0] 1787366038.739541 trunc from 0x5b0000 to 0x2615e1 (0x34ea1e bytes) 000067[0] 1787366038.822714 mapread 0x11e000 thru 0x1210bf (0x30c0 bytes) 000068[0] 1787366038.841877 mapread 0x129000 thru 0x129da0 (0xda1 bytes) 000069[0] 1787366038.879457 trunc from 0x2615e2 to 0x711864 (0x4b0283 bytes) 000070[0] 1787366038.935183 read 0x440000 thru 0x44b5c8 (0xb5c9 bytes) 000072[1] 1787366038.970583 mapread 0x573000 thru 0x576757 (0x3758 bytes) 000074[0] 1787366039.056458 trunc from 0x711865 to 0x0cea36 (0x642e2e bytes) 000075[1] 1787366039.122199 mapread 0x017000 thru 0x022748 (0xb749 bytes) 000077[1] 1787366039.216941 trunc from 0x0cea37 to 0x7b0320 (0x6e18ea bytes) 000079[1] 1787366039.354781 mapread 0x739000 thru 0x7398f8 (0x8f9 bytes) 000080[0] 1787366039.372649 trunc from 0x7b0321 to 0x294fbb (0x51b365 bytes) 000083[0] 1787366039.494074 read 0x105000 thru 0x105cc5 (0xcc6 bytes) 000084[1] 1787366039.603025 mapread 0x04e000 thru 0x055b95 (0x7b96 bytes) truncating to largest ever: 0x96663c 000085[0] 1787366039.662878 trunc from 0x294fbc to 0x96663b (0x6d1680 bytes) 000086[1] 1787366039.710216 mapwrite 0x952000 thru 0x95d92b (0xb92c bytes) 000087[1] 1787366039.926584 write_OD 0x1b8000 thru 0x1c6fff (0xf000 bytes) 000088[1] 1787366040.024342 trunc from 0x96663c to 0x5882bd (0x3de37e bytes) 000089[0] 1787366040.112057 read 0x087000 thru 0x0952a7 (0xe2a8 bytes) 000090[0] 1787366040.156014 write 0x072000 thru 0x081982 (0xf983 bytes) 000091[0] 1787366040.176830 mapread 0x260000 thru 0x26f31b (0xf31c bytes) 000093[0] 1787366040.252799 write 0x349000 thru 0x354216 (0xb217 bytes) 000094[0] 1787366040.276458 trunc from 0x5882be to 0x55d0e0 (0x2b1dd bytes) 000096[0] 1787366040.313955 mapread 0x10c000 thru 0x11b644 (0xf645 bytes) 000098[1] 1787366040.360795 mapread 0x081000 thru 0x0860fe (0x50ff bytes) 000102[1] 1787366040.421444 trunc from 0x55d0e1 to 0x5711ae (0x140ce bytes) 000103[1] 1787366040.474542 mapwrite 0x23e000 thru 0x24dd3b (0xfd3c bytes) 000106[1] 1787366040.675001 mapwrite 0x605000 thru 0x60a3e8 (0x53e9 bytes) 000107[1] 1787366040.863242 mapread 0x4e1000 thru 0x4e594c (0x494d bytes) 000108[0] 1787366040.883705 write 0x9e1000 thru 0x9f0636 (0xf637 bytes) 000110[0] 1787366040.907607 mapwrite 0x027000 thru 0x02f3b1 (0x83b2 bytes) 000111[1] 1787366041.001616 write_OD 0x417000 thru 0x41efff (0x8000 bytes) 000112[1] 1787366041.053979 write_OD 0x5dc000 thru 0x5e7fff (0xc000 bytes) 000113[1] 1787366041.122674 trunc from 0x9f0637 to 0x4da3cf (0x516267 bytes) 000115[1] 1787366041.171085 write_OD 0x606000 thru 0x608fff (0x3000 bytes) 000116[0] 1787366041.233413 mapwrite 0x8ed000 thru 0x8f7562 (0xa563 bytes) 000120[1] 1787366041.405469 read_OD 0x0ad000 thru 0x0b2fff (0x6000 bytes) 000122[0] 1787366041.441019 read 0x326000 thru 0x32a8a3 (0x48a4 bytes) 000125[0] 1787366041.451671 trunc from 0x8f7563 to 0x8dd64a (0x19f18 bytes) 000126[0] 1787366041.488763 read 0x4a9000 thru 0x4b48de (0xb8df bytes) 000127[1] 1787366041.519425 read_OD 0x59b000 thru 0x5a9fff (0xf000 bytes) 000128[0] 1787366041.558554 mapwrite 0x9e9000 thru 0x9f6063 (0xd064 bytes) 000129[0] 1787366041.720554 write 0x238000 thru 0x23eb63 (0x6b64 bytes) 000130[0] 1787366041.733451 write 0x026000 thru 0x02e6d1 (0x86d2 bytes) 000131[0] 1787366041.744518 mapwrite 0x923000 thru 0x9238c9 (0x8ca bytes) 000134[0] 1787366041.941746 mapread 0x14d000 thru 0x15a10d (0xd10e bytes) 000138[0] 1787366042.027823 mapread 0x516000 thru 0x51d041 (0x7042 bytes) 000139[0] 1787366042.087094 mapread 0x552000 thru 0x5557f2 (0x37f3 bytes) 000140[0] 1787366042.130887 write 0x92b000 thru 0x935843 (0xa844 bytes) 000142[0] 1787366042.163436 mapwrite 0x2fe000 thru 0x2ffb40 (0x1b41 bytes) 000144[0] 1787366042.246616 read 0x486000 thru 0x48f0ec (0x90ed bytes) 000148[0] 1787366042.285087 mapread 0x969000 thru 0x96f231 (0x6232 bytes) 000151[0] 1787366042.344529 mapwrite 0x10d000 thru 0x112d4b (0x5d4c bytes) 000157[1] 1787366042.536487 write_OD 0x094000 thru 0x0a1fff (0xe000 bytes) 000158[0] 1787366042.606416 write 0x566000 thru 0x56d510 (0x7511 bytes) 000160[1] 1787366042.642689 trunc from 0x9f6064 to 0x1973b0 (0x85ecb3 bytes) 000162[0] 1787366042.741575 read 0x128000 thru 0x1363da (0xe3db bytes) 000166[0] 1787366042.826794 mapwrite 0x8af000 thru 0x8b0a22 (0x1a23 bytes) 000169[1] 1787366043.239635 write_OD 0x8fa000 thru 0x905fff (0xc000 bytes) 000170[1] 1787366043.368036 read_OD 0x453000 thru 0x457fff (0x5000 bytes) 000172[1] 1787366043.380664 mapread 0x43b000 thru 0x44708e (0xc08f bytes) 000176[0] 1787366043.396216 mapwrite 0x266000 thru 0x26c570 (0x6571 bytes) 000179[0] 1787366043.514050 write 0x334000 thru 0x33888b (0x488c bytes) 000183[0] 1787366043.578397 mapwrite 0x16a000 thru 0x171dc1 (0x7dc2 bytes) 000186[0] 1787366043.712499 trunc from 0x906000 to 0x7fb809 (0x10a7f6 bytes) 000190[1] 1787366043.776737 mapread 0x59e000 thru 0x5ac75d (0xe75e bytes) 000191[1] 1787366043.851558 mapwrite 0x7fa000 thru 0x7ff48f (0x5490 bytes) 000195[0] 1787366044.133010 read 0x3b8000 thru 0x3bb884 (0x3885 bytes) 000196[0] 1787366044.161478 mapwrite 0x0f5000 thru 0x1020e6 (0xd0e7 bytes) 000197[1] 1787366044.305095 write_OD 0x5da000 thru 0x5dcfff (0x3000 bytes) 000201[0] 1787366044.356080 write 0x970000 thru 0x9710c1 (0x10c2 bytes) 000204[0] 1787366044.410342 mapwrite 0x1fc000 thru 0x1fc4fd (0x4fe bytes) 000205[1] 1787366044.512330 read_OD 0x580000 thru 0x581fff (0x2000 bytes) 000206[1] 1787366044.554944 write_OD 0x31d000 thru 0x31dfff (0x1000 bytes) 000207[0] 1787366044.640294 read 0x40b000 thru 0x41a20d (0xf20e bytes) 000209[1] 1787366044.666792 write_OD 0x213000 thru 0x21afff (0x8000 bytes) 000212[0] 1787366044.765346 mapread 0x034000 thru 0x03ff3b (0xbf3c bytes) 000213[1] 1787366044.791739 mapwrite 0x72d000 thru 0x72ff1d (0x2f1e bytes) 000214[0] 1787366044.873574 write 0x583000 thru 0x58ef66 (0xbf67 bytes) 000215[1] 1787366044.893664 mapwrite 0x1fc000 thru 0x2092f4 (0xd2f5 bytes) 000216[0] 1787366045.010541 read 0x27b000 thru 0x27ebf1 (0x3bf2 bytes) 000218[1] 1787366045.029539 write_OD 0x113000 thru 0x114fff (0x2000 bytes) 000219[1] 1787366045.085931 write_OD 0x566000 thru 0x566fff (0x1000 bytes) 000221[0] 1787366045.164555 read 0x59e000 thru 0x5a360a (0x560b bytes) 000224[1] 1787366045.174486 mapread 0x4d7000 thru 0x4e5062 (0xe063 bytes) 000225[0] 1787366045.233518 write 0x694000 thru 0x6a3c0e (0xfc0f bytes) 000226[0] 1787366045.267810 trunc from 0x9710c2 to 0x91a859 (0x56868 bytes) 000228[1] 1787366045.346988 write_OD 0x960000 thru 0x963fff (0x4000 bytes) 000231[1] 1787366045.437941 write_OD 0x5e8000 thru 0x5e8fff (0x1000 bytes) 000232[0] 1787366045.516278 read 0x6e3000 thru 0x6e8cf7 (0x5cf8 bytes) 000233[1] 1787366045.531527 trunc from 0x964000 to 0x71d8e0 (0x24671f bytes) 000235[0] 1787366045.596183 mapread 0x458000 thru 0x459942 (0x1943 bytes) 000237[1] 1787366045.658754 read_OD 0x29e000 thru 0x29efff (0x1000 bytes) 000239[1] 1787366045.671803 mapread 0x49c000 thru 0x4a017f (0x4180 bytes) 000240[0] 1787366045.697431 trunc from 0x71d8e1 to 0x0f616c (0x627774 bytes) 000241[0] 1787366045.758162 read 0x036000 thru 0x03ed54 (0x8d55 bytes) 000243[0] 1787366045.814172 read 0x0e1000 thru 0x0e61ec (0x51ed bytes) 000244[0] 1787366045.818360 trunc from 0x0f616d to 0x398da7 (0x2a2c3b bytes) 000246[0] 1787366045.840816 trunc from 0x398da8 to 0x7bccd2 (0x423f2b bytes) 000249[0] 1787366045.876430 write 0x8a7000 thru 0x8b368c (0xc68d bytes) 000250[1] 1787366045.877953 read_OD 0x21d000 thru 0x227fff (0xb000 bytes) 000251[0] 1787366045.932522 mapwrite 0x0ff000 thru 0x10ea91 (0xfa92 bytes) 000252[0] 1787366046.101095 trunc from 0x8b368d to 0x25ac43 (0x658a49 bytes) 000253[1] 1787366046.158818 trunc from 0x25ac44 to 0x8cd4c6 (0x672883 bytes) 000257[0] 1787366046.322819 read 0x1a6000 thru 0x1a8b78 (0x2b79 bytes) 000258[0] 1787366046.344040 mapwrite 0x017000 thru 0x023152 (0xc153 bytes) 000262[1] 1787366046.537914 read_OD 0x03e000 thru 0x047fff (0xa000 bytes) 000263[0] 1787366046.569923 write 0x4dc000 thru 0x4e3a46 (0x7a47 bytes) 000267[0] 1787366046.671031 read 0x5a9000 thru 0x5b8d39 (0xfd3a bytes) 000271[1] 1787366046.689094 mapread 0x356000 thru 0x360313 (0xa314 bytes) 000273[0] 1787366046.770469 read 0x543000 thru 0x54c71a (0x971b bytes) 000279[0] 1787366046.783446 mapwrite 0x2a0000 thru 0x2ab55f (0xb560 bytes) 000280[0] 1787366046.903411 mapwrite 0x38e000 thru 0x38e6b7 (0x6b8 bytes) 000281[1] 1787366047.019601 read_OD 0x87f000 thru 0x882fff (0x4000 bytes) 000282[0] 1787366047.082080 write 0x7e9000 thru 0x7f3182 (0xa183 bytes) 000284[1] 1787366047.132422 mapread 0x6c6000 thru 0x6d01b0 (0xa1b1 bytes) 000288[0] 1787366047.200629 mapwrite 0x3ba000 thru 0x3c63bc (0xc3bd bytes) 000293[1] 1787366047.449545 mapread 0x828000 thru 0x82c527 (0x4528 bytes) 000295[0] 1787366047.495511 mapread 0x0e4000 thru 0x0edc22 (0x9c23 bytes) 000298[0] 1787366047.514802 read 0x211000 thru 0x217c88 (0x6c89 bytes) 000302[1] 1787366047.526726 mapwrite 0x57b000 thru 0x585b79 (0xab7a bytes) 000305[1] 1787366047.627756 mapread 0x316000 thru 0x31bac9 (0x5aca bytes) 000306[1] 1787366047.662339 mapwrite 0x1e5000 thru 0x1f14de (0xc4df bytes) 000308[0] 1787366047.745043 read 0x468000 thru 0x46ae64 (0x2e65 bytes) 000309[0] 1787366047.764685 mapread 0x177000 thru 0x180b48 (0x9b49 bytes) 000310[0] 1787366047.783853 mapwrite 0x514000 thru 0x5224fd (0xe4fe bytes) 000311[0] 1787366047.888623 read 0x46d000 thru 0x47a458 (0xd459 bytes) 000312[0] 1787366047.919322 mapwrite 0x7b6000 thru 0x7c339e (0xd39f bytes) 000315[1] 1787366048.016368 read_OD 0x29a000 thru 0x2a2fff (0x9000 bytes) 000316[1] 1787366048.041477 mapwrite 0x559000 thru 0x566aa8 (0xdaa9 bytes) 000317[0] 1787366048.167699 write 0x290000 thru 0x29f176 (0xf177 bytes) 000318[1] 1787366048.199561 mapread 0x054000 thru 0x061a4d (0xda4e bytes) 000319[0] 1787366048.276664 trunc from 0x8cd4c7 to 0x5a1e4e (0x32b678 bytes) 000321[0] 1787366048.330978 write 0x950000 thru 0x95d0c2 (0xd0c3 bytes) 000322[0] 1787366048.334505 mapwrite 0x808000 thru 0x80b3f2 (0x33f3 bytes) 000323[1] 1787366048.402392 trunc from 0x95d0c3 to 0x3d9e49 (0x583279 bytes) 000326[1] 1787366048.486193 mapwrite 0x673000 thru 0x67523f (0x2240 bytes) 000329[0] 1787366048.585775 mapread 0x09f000 thru 0x0a5739 (0x673a bytes) 000331[0] 1787366048.624341 mapread 0x426000 thru 0x435408 (0xf409 bytes) 000332[0] 1787366048.649485 mapwrite 0x70c000 thru 0x71bed0 (0xfed1 bytes) 000333[1] 1787366048.787923 mapread 0x58e000 thru 0x591937 (0x3938 bytes) 000334[1] 1787366048.839721 mapwrite 0x361000 thru 0x361bb8 (0xbb9 bytes) 000337[0] 1787366049.008002 mapread 0x552000 thru 0x558895 (0x6896 bytes) 000339[1] 1787366049.103431 read_OD 0x64e000 thru 0x65bfff (0xe000 bytes) 000347[0] 1787366049.136614 trunc from 0x71bed1 to 0x2118c4 (0x50a60c bytes) 000349[0] 1787366049.287571 trunc from 0x2118c5 to 0x91d78f (0x70becb bytes) truncating to largest ever: 0x975d4a 000350[1] 1787366049.318369 trunc from 0x91d790 to 0x975d49 (0x585ba bytes) 000351[0] 1787366049.361015 trunc from 0x975d4a to 0x01aaf7 (0x95b252 bytes) 000352[1] 1787366049.442206 mapwrite 0x389000 thru 0x38e9cb (0x59cc bytes) 000353[0] 1787366049.695809 read 0x1e4000 thru 0x1eb2e7 (0x72e8 bytes) 000354[1] 1787366049.901544 trunc from 0x38e9cc to 0x2761a1 (0x11882a bytes) 000357[0] 1787366050.003476 mapread 0x250000 thru 0x253f20 (0x3f21 bytes) 000362[1] 1787366050.092153 trunc from 0x2761a2 to 0x75cd7b (0x4e6bda bytes) 000364[1] 1787366050.160154 trunc from 0x75cd7c to 0x27cd52 (0x4e0029 bytes) 000365[1] 1787366050.191669 mapwrite 0x2f5000 thru 0x2fc210 (0x7211 bytes) 000367[1] 1787366050.413944 mapwrite 0x352000 thru 0x35b2ed (0x92ee bytes) 000368[0] 1787366050.583506 write 0x6c6000 thru 0x6c81b9 (0x21ba bytes) 000375[0] 1787366050.840351 mapwrite 0x102000 thru 0x111e8c (0xfe8d bytes) 000378[0] 1787366051.091021 trunc from 0x6c81ba to 0x822bce (0x15aa15 bytes) 000379[1] 1787366051.127936 trunc from 0x822bcf to 0x11bd5b (0x706e73 bytes) 000380[0] 1787366051.158057 write 0x54a000 thru 0x54ab71 (0xb72 bytes) 000385[1] 1787366051.267205 trunc from 0x54ab72 to 0x0cda26 (0x47d14b bytes) 000387[0] 1787366051.319692 trunc from 0x0cda27 to 0x10c9f9 (0x3efd3 bytes) 000388[0] 1787366051.356028 trunc from 0x10c9fa to 0x44a997 (0x33df9e bytes) 000389[0] 1787366051.393056 write 0x96a000 thru 0x9710ec (0x70ed bytes) 000390[0] 1787366051.409468 mapread 0x8fa000 thru 0x903c0a (0x9c0b bytes) 000392[0] 1787366051.546764 mapread 0x5cc000 thru 0x5da69a (0xe69b bytes) 000395[1] 1787366051.568600 mapwrite 0x222000 thru 0x2298b6 (0x78b7 bytes) 000397[0] 1787366051.799490 mapwrite 0x8db000 thru 0x8df73f (0x4740 bytes) 000399[0] 1787366052.016783 mapwrite 0x4e3000 thru 0x4efd17 (0xcd18 bytes) 000401[1] 1787366052.292491 read_OD 0x4fe000 thru 0x503fff (0x6000 bytes) 000403[1] 1787366052.330090 mapread 0x002000 thru 0x0082f4 (0x62f5 bytes) 000404[0] 1787366052.351911 mapread 0x58b000 thru 0x591a3c (0x6a3d bytes) 000406[1] 1787366052.394758 trunc from 0x9710ed to 0x7728de (0x1fe80e bytes) 000407[0] 1787366052.433764 mapwrite 0x951000 thru 0x958c64 (0x7c65 bytes) 000408[0] 1787366052.564347 trunc from 0x958c65 to 0x291abc (0x6c71a8 bytes) 000411[0] 1787366052.610959 write 0x9dc000 thru 0x9de865 (0x2866 bytes) 000412[1] 1787366052.627907 write_OD 0x979000 thru 0x984fff (0xc000 bytes) 000414[1] 1787366052.694472 read_OD 0x8fe000 thru 0x909fff (0xc000 bytes) 000421[0] 1787366052.703687 write 0x9b1000 thru 0x9b3943 (0x2944 bytes) 000425[1] 1787366052.776052 read_OD 0x30a000 thru 0x313fff (0xa000 bytes) 000429[1] 1787366052.804528 write_OD 0x8d0000 thru 0x8defff (0xf000 bytes) 000431[0] 1787366052.851746 mapread 0x72c000 thru 0x737339 (0xb33a bytes) 000435[1] 1787366052.948838 mapread 0x849000 thru 0x85801b (0xf01c bytes) 000436[1] 1787366052.996322 read_OD 0x5ac000 thru 0x5b1fff (0x6000 bytes) 000437[1] 1787366053.003960 trunc from 0x9de866 to 0x778bdd (0x265c88 bytes) 000439[0] 1787366053.056091 write 0x8a3000 thru 0x8b1eaf (0xeeb0 bytes) 000440[0] 1787366053.099349 mapread 0x008000 thru 0x00be7b (0x3e7c bytes) 000441[0] 1787366053.140529 write 0x7b8000 thru 0x7c78c2 (0xf8c3 bytes) 000442[1] 1787366053.150943 mapwrite 0x87d000 thru 0x87d8bf (0x8c0 bytes) 000443[1] 1787366053.247544 mapwrite 0x97d000 thru 0x97ef22 (0x1f23 bytes) 000446[1] 1787366053.347386 trunc from 0x97ef23 to 0x318401 (0x666b21 bytes) 000449[1] 1787366053.389386 mapwrite 0x07d000 thru 0x089463 (0xc464 bytes) 000450[0] 1787366053.469803 mapwrite 0x282000 thru 0x285f43 (0x3f44 bytes) 000451[1] 1787366053.642399 write_OD 0x212000 thru 0x220fff (0xf000 bytes) 000452[1] 1787366053.751051 mapwrite 0x233000 thru 0x238f64 (0x5f65 bytes) 000453[1] 1787366053.854080 read_OD 0x22e000 thru 0x237fff (0xa000 bytes) 000454[0] 1787366053.886976 mapwrite 0x173000 thru 0x17e214 (0xb215 bytes) 000457[1] 1787366054.068757 write_OD 0x6ef000 thru 0x6f6fff (0x8000 bytes) 000458[1] 1787366054.141387 trunc from 0x6f7000 to 0x231cb8 (0x4c5347 bytes) 000460[1] 1787366054.211746 write_OD 0x0e7000 thru 0x0e7fff (0x1000 bytes) 000461[1] 1787366054.287304 mapwrite 0x3fd000 thru 0x403d8d (0x6d8e bytes) 000463[1] 1787366054.610701 trunc from 0x403d8e to 0x84f2ca (0x44b53d bytes) 000464[1] 1787366054.638949 mapwrite 0x584000 thru 0x58537b (0x137c bytes) 000467[0] 1787366054.930053 mapread 0x0f9000 thru 0x101dba (0x8dbb bytes) 000469[1] 1787366055.159446 write_OD 0x225000 thru 0x233fff (0xf000 bytes) 000471[1] 1787366055.272348 mapread 0x484000 thru 0x4916c7 (0xd6c8 bytes) 000472[1] 1787366055.378279 write_OD 0x468000 thru 0x474fff (0xd000 bytes) 000474[0] 1787366055.489346 mapread 0x623000 thru 0x624fe1 (0x1fe2 bytes) 000475[1] 1787366055.520293 mapwrite 0x483000 thru 0x48fe43 (0xce44 bytes) 000483[0] 1787366055.893601 mapwrite 0x4f4000 thru 0x5035dc (0xf5dd bytes) 000486[1] 1787366056.218600 mapread 0x7bb000 thru 0x7c582b (0xa82c bytes) 000488[1] 1787366056.337040 mapread 0x7f8000 thru 0x801e25 (0x9e26 bytes) 000489[0] 1787366056.433690 mapread 0x670000 thru 0x67bfdf (0xbfe0 bytes) 000491[0] 1787366056.531350 trunc from 0x84f2cb to 0x7d9d27 (0x755a3 bytes) 000492[1] 1787366056.597407 trunc from 0x7d9d28 to 0x0e2122 (0x6f7c05 bytes) 000493[0] 1787366056.637985 write 0x9c0000 thru 0x9c60b7 (0x60b8 bytes) 000494[1] 1787366056.680461 write_OD 0x377000 thru 0x37ffff (0x9000 bytes) 000495[0] 1787366056.767975 write 0x575000 thru 0x58283c (0xd83d bytes) 000497[1] 1787366056.856955 trunc from 0x9c60b8 to 0x600f80 (0x3c5137 bytes) 000498[1] 1787366056.945980 read_OD 0x0ef000 thru 0x0f8fff (0xa000 bytes) 000500[1] 1787366056.973643 mapwrite 0x44c000 thru 0x44c29d (0x29e bytes) All operations completed A-OK! PASS 16a (56s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 22:34:23 (1787366063) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.44078 s, 19.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.459387 s, 18.3 MB/s Chance of close/open is 1 in 50 Seed set to 6066 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[0] 1787366067.228418 write 0x3c9566 thru 0x3cb488 (0x1f23 bytes) truncating to largest ever: 0x9c9d0c 000006[1] 1787366067.303380 trunc from 0x3cb489 to 0x9c9d0b (0x5fe883 bytes) 000009[0] 1787366067.436991 mapwrite 0x1df4db thru 0x1e0ac9 (0x15ef bytes) 000010[0] 1787366067.609024 mapread 0x269ba7 thru 0x26a7cd (0xc27 bytes) 000011[1] 1787366067.711320 read 0x779cb8 thru 0x77b590 (0x18d9 bytes) 000013[1] 1787366067.798356 trunc from 0x9c9d0c to 0x25d3d6 (0x76c935 bytes) 000016[1] 1787366067.902623 mapread 0x0af377 thru 0x0af400 (0x8a bytes) 000017[1] 1787366067.937654 read 0x004fdf thru 0x005fa4 (0xfc6 bytes) 000018[0] 1787366068.028454 mapwrite 0x08806f thru 0x088445 (0x3d7 bytes) 000019[1] 1787366068.128951 mapwrite 0x3e6185 thru 0x3e773f (0x15bb bytes) 000020[1] 1787366068.247314 trunc from 0x3e7740 to 0x7b80f1 (0x3d09b2 bytes) 000024[0] 1787366068.286602 trunc from 0x7b80f2 to 0x957a14 (0x19f923 bytes) 000026[1] 1787366068.373752 mapwrite 0x3b7b07 thru 0x3b7b78 (0x72 bytes) 000029[1] 1787366068.490832 trunc from 0x957a15 to 0x2b8a20 (0x69eff4 bytes) 000030[1] 1787366068.527043 write 0x6d36c1 thru 0x6d3fdd (0x91d bytes) 000033[0] 1787366068.561880 write 0x912a66 thru 0x913cd0 (0x126b bytes) 000034[1] 1787366068.594399 read 0x3ecfa7 thru 0x3ed7ed (0x847 bytes) 000035[1] 1787366068.620330 write 0x08b44e thru 0x08bffc (0xbaf bytes) 000037[0] 1787366068.639313 mapwrite 0x9f89eb thru 0x9f89ef (0x5 bytes) 000039[0] 1787366068.840660 trunc from 0x9f89f0 to 0x023ed3 (0x9d4b1c bytes) 000041[0] 1787366068.913223 write 0x253050 thru 0x254954 (0x1905 bytes) 000042[1] 1787366068.915955 trunc from 0x254955 to 0x381763 (0x12ce0f bytes) 000043[1] 1787366068.975772 trunc from 0x381764 to 0x457434 (0xd5cd1 bytes) 000045[0] 1787366069.020617 mapread 0x025f3b thru 0x027db1 (0x1e77 bytes) 000046[0] 1787366069.068672 write 0x7c17c7 thru 0x7c2594 (0xdce bytes) 000047[0] 1787366069.082055 write 0x8b6e3b thru 0x8b7fd5 (0x119b bytes) 000048[1] 1787366069.118557 write 0x843266 thru 0x844bcf (0x196a bytes) 000049[0] 1787366069.166827 read 0x6816ad thru 0x682406 (0xd5a bytes) 000050[1] 1787366069.187474 read 0x1bccc0 thru 0x1bd8c7 (0xc08 bytes) 000051[1] 1787366069.207039 mapread 0x25e321 thru 0x25ea80 (0x760 bytes) 000054[1] 1787366069.276213 trunc from 0x8b7fd6 to 0x1ce11e (0x6e9eb7 bytes) 000056[0] 1787366069.350539 mapwrite 0x285424 thru 0x285c15 (0x7f2 bytes) 000058[1] 1787366069.560874 trunc from 0x285c16 to 0x55d8ec (0x2d7cd7 bytes) 000060[1] 1787366069.650027 mapwrite 0x9563a8 thru 0x956d8a (0x9e3 bytes) 000061[1] 1787366069.748970 mapwrite 0x231ce4 thru 0x232100 (0x41d bytes) 000065[1] 1787366069.999606 mapwrite 0x9bb44d thru 0x9bbcf3 (0x8a7 bytes) 000066[0] 1787366070.137866 read 0x04c89c thru 0x04c9db (0x140 bytes) 000069[1] 1787366070.249603 read 0x2c1f11 thru 0x2c20f7 (0x1e7 bytes) 000073[0] 1787366070.298508 write 0x583988 thru 0x583df1 (0x46a bytes) 000074[1] 1787366070.335932 mapwrite 0x2702b6 thru 0x271146 (0xe91 bytes) 000075[1] 1787366070.439000 mapread 0x8b81b5 thru 0x8b97fa (0x1646 bytes) 000076[0] 1787366070.487581 mapread 0x9ab1cb thru 0x9aca96 (0x18cc bytes) 000078[0] 1787366070.560373 mapread 0x6d2422 thru 0x6d3a89 (0x1668 bytes) 000080[0] 1787366070.625728 read 0x3986b6 thru 0x399b0b (0x1456 bytes) 000081[1] 1787366070.633082 write 0x144dac thru 0x145edf (0x1134 bytes) 000084[0] 1787366070.758482 mapwrite 0x995763 thru 0x996207 (0xaa5 bytes) 000087[1] 1787366070.979580 mapread 0x3c60ef thru 0x3c7358 (0x126a bytes) 000089[1] 1787366071.045649 trunc from 0x9bbcf4 to 0x57b39c (0x440957 bytes) 000092[0] 1787366071.198206 trunc from 0x57b39d to 0x896e75 (0x31bad9 bytes) 000097[1] 1787366071.419968 mapwrite 0x93bca0 thru 0x93d8e8 (0x1c49 bytes) 000099[0] 1787366071.687712 mapread 0x19a8b9 thru 0x19a9dc (0x124 bytes) 000100[1] 1787366071.750012 mapread 0x8b2ac1 thru 0x8b3321 (0x861 bytes) 000103[0] 1787366071.826028 write 0x983451 thru 0x983a7b (0x62b bytes) 000107[1] 1787366071.971942 write 0x70dec7 thru 0x70e6da (0x814 bytes) 000109[0] 1787366072.055705 read 0x34df25 thru 0x34df96 (0x72 bytes) 000111[1] 1787366072.114355 write 0x956c34 thru 0x9585ce (0x199b bytes) 000116[1] 1787366072.247977 trunc from 0x983a7c to 0x8c10a8 (0xc29d3 bytes) 000119[1] 1787366072.361369 mapread 0x443637 thru 0x4452c2 (0x1c8c bytes) 000120[0] 1787366072.426466 trunc from 0x8c10a9 to 0x4ae44c (0x412c5c bytes) 000122[1] 1787366072.608871 mapread 0x4a3f9c thru 0x4a5b39 (0x1b9e bytes) 000123[1] 1787366072.661826 mapread 0x309583 thru 0x309f72 (0x9f0 bytes) 000125[0] 1787366072.728900 write 0x65ea81 thru 0x6607a4 (0x1d24 bytes) 000130[1] 1787366072.809225 mapwrite 0x049374 thru 0x04a41f (0x10ac bytes) 000137[0] 1787366073.076503 mapwrite 0x5a8fb6 thru 0x5aa277 (0x12c2 bytes) 000139[0] 1787366073.219799 read 0x0d2907 thru 0x0d4160 (0x185a bytes) 000140[0] 1787366073.266458 mapread 0x46c07f thru 0x46c56f (0x4f1 bytes) 000141[1] 1787366073.284915 mapread 0x0b28c9 thru 0x0b3018 (0x750 bytes) 000143[1] 1787366073.343594 read 0x150ded thru 0x151cd8 (0xeec bytes) 000144[0] 1787366073.362986 trunc from 0x6607a5 to 0x0a6a2e (0x5b9d76 bytes) 000145[1] 1787366073.423495 write 0x77ac73 thru 0x77c44c (0x17da bytes) 000147[0] 1787366073.474232 mapread 0x097c1b thru 0x098034 (0x41a bytes) 000148[1] 1787366073.506240 write 0x70ab66 thru 0x70b260 (0x6fb bytes) 000149[0] 1787366073.529638 read 0x401990 thru 0x4032fb (0x196c bytes) 000150[0] 1787366073.582357 read 0x26778e thru 0x2689df (0x1252 bytes) 000152[1] 1787366073.666501 write 0x4a405a thru 0x4a55b1 (0x1558 bytes) 000153[1] 1787366073.744837 read 0x33cd73 thru 0x33d707 (0x995 bytes) 000155[0] 1787366073.807887 mapwrite 0x42c22a thru 0x42dbb1 (0x1988 bytes) 000156[0] 1787366073.965910 mapread 0x4b7efb thru 0x4b9578 (0x167e bytes) 000158[1] 1787366074.044067 read 0x31c66d thru 0x31dbae (0x1542 bytes) 000159[1] 1787366074.081578 write 0x8b7435 thru 0x8b8d58 (0x1924 bytes) 000160[0] 1787366074.155099 trunc from 0x8b8d59 to 0x120275 (0x798ae3 bytes) 000162[0] 1787366074.256229 mapread 0x033eef thru 0x034beb (0xcfd bytes) 000163[1] 1787366074.270228 trunc from 0x120276 to 0x64072d (0x5204b8 bytes) 000165[1] 1787366074.377403 mapread 0x28410c thru 0x2855f9 (0x14ee bytes) 000169[1] 1787366074.475821 read 0x2c8860 thru 0x2c9f0a (0x16ab bytes) 000172[1] 1787366074.518060 mapwrite 0x79b518 thru 0x79ca3f (0x1528 bytes) 000174[0] 1787366074.626374 write 0x5ef7ab thru 0x5f08d4 (0x112a bytes) 000175[1] 1787366074.663773 read 0x132281 thru 0x13241d (0x19d bytes) 000176[0] 1787366074.681640 trunc from 0x79ca40 to 0x2015e8 (0x59b457 bytes) 000178[1] 1787366074.717838 read 0x0f6d85 thru 0x0f70bb (0x337 bytes) 000179[0] 1787366074.756569 write 0x44ea38 thru 0x450427 (0x19f0 bytes) 000181[1] 1787366074.827872 mapwrite 0x331ec8 thru 0x3324f6 (0x62f bytes) 000182[1] 1787366075.057879 mapwrite 0x1648ee thru 0x1664c3 (0x1bd6 bytes) 000185[1] 1787366075.202324 read 0x42887d thru 0x4292fb (0xa7f bytes) 000186[0] 1787366075.225746 write 0x95d91b thru 0x95ecd6 (0x13bc bytes) 000187[0] 1787366075.243853 write 0x6970e8 thru 0x697253 (0x16c bytes) 000188[0] 1787366075.266923 read 0x4238f3 thru 0x4244e7 (0xbf5 bytes) 000189[1] 1787366075.273532 mapwrite 0x1790bc thru 0x17a12b (0x1070 bytes) 000191[1] 1787366075.525540 read 0x13f786 thru 0x14135e (0x1bd9 bytes) 000192[1] 1787366075.557650 read 0x7f9126 thru 0x7f971a (0x5f5 bytes) 000195[0] 1787366075.650943 mapread 0x6c80a9 thru 0x6c8f2d (0xe85 bytes) 000199[0] 1787366075.843519 read 0x15c5ca thru 0x15c818 (0x24f bytes) 000200[0] 1787366075.887076 write 0x873883 thru 0x873a1b (0x199 bytes) 000206[0] 1787366075.985656 read 0x81861c thru 0x818cfd (0x6e2 bytes) 000208[0] 1787366075.991737 mapread 0x21c6b3 thru 0x21e3cf (0x1d1d bytes) 000209[1] 1787366076.026430 write 0x69978f thru 0x69b459 (0x1ccb bytes) 000216[0] 1787366076.291845 read 0x88f027 thru 0x88f6ae (0x688 bytes) 000220[1] 1787366076.420302 mapread 0x2fc4d2 thru 0x2fcb7a (0x6a9 bytes) 000231[0] 1787366076.520598 trunc from 0x95ecd7 to 0x19548c (0x7c984a bytes) 000232[1] 1787366076.593962 mapwrite 0x69341d thru 0x69538e (0x1f72 bytes) 000236[0] 1787366076.740378 trunc from 0x69538f to 0x7ccf50 (0x137bc2 bytes) 000237[0] 1787366076.816720 write 0x086dd8 thru 0x0878b0 (0xad9 bytes) 000238[1] 1787366076.851320 trunc from 0x7ccf51 to 0x009e0f (0x7c3141 bytes) 000240[0] 1787366076.909077 mapread 0x00873c thru 0x008f6e (0x833 bytes) 000243[0] 1787366076.997871 mapwrite 0x358a8d thru 0x35959f (0xb13 bytes) 000244[0] 1787366077.184696 mapread 0x2a7ae0 thru 0x2a81e8 (0x709 bytes) 000245[1] 1787366077.231998 write 0x0bb43e thru 0x0bbe34 (0x9f7 bytes) 000247[0] 1787366077.294020 write 0x0f03a5 thru 0x0f1d10 (0x196c bytes) 000251[1] 1787366077.362472 mapwrite 0x0b93ff thru 0x0bb0b3 (0x1cb5 bytes) 000256[0] 1787366077.603228 write 0x30f288 thru 0x30fba9 (0x922 bytes) 000259[0] 1787366077.664871 mapwrite 0x1ec6f7 thru 0x1ee405 (0x1d0f bytes) 000260[0] 1787366077.763602 read 0x039cf9 thru 0x03aa15 (0xd1d bytes) 000261[0] 1787366077.816927 trunc from 0x3595a0 to 0x34f180 (0xa41f bytes) 000265[1] 1787366077.936603 mapwrite 0x950450 thru 0x9520b0 (0x1c61 bytes) 000273[0] 1787366078.218534 mapread 0x4c04c3 thru 0x4c0b31 (0x66f bytes) 000274[0] 1787366078.268908 mapread 0x181096 thru 0x18140c (0x377 bytes) 000276[0] 1787366078.338578 read 0x6b6c2d thru 0x6b8292 (0x1666 bytes) 000277[1] 1787366078.344728 read 0x939918 thru 0x939a7f (0x168 bytes) 000278[1] 1787366078.351865 trunc from 0x9520b1 to 0x6e3a7b (0x26e635 bytes) 000279[1] 1787366078.418306 mapread 0x61b97a thru 0x61cb02 (0x1189 bytes) 000280[0] 1787366078.432557 mapread 0x0263eb thru 0x027ce0 (0x18f6 bytes) 000283[0] 1787366078.546556 mapwrite 0x4beed8 thru 0x4bef5f (0x88 bytes) 000284[1] 1787366078.737293 write 0x20258c thru 0x202f2c (0x9a1 bytes) 000290[0] 1787366078.913588 mapread 0x20859c thru 0x208e06 (0x86b bytes) 000291[1] 1787366078.963507 mapwrite 0x681344 thru 0x681995 (0x652 bytes) 000292[0] 1787366079.065850 mapread 0x26883b thru 0x26a200 (0x19c6 bytes) 000296[1] 1787366079.177266 mapwrite 0x3a6fd9 thru 0x3a7851 (0x879 bytes) 000298[1] 1787366079.353204 read 0x6b32f3 thru 0x6b42e4 (0xff2 bytes) 000299[1] 1787366079.390958 mapwrite 0x46815a thru 0x468c79 (0xb20 bytes) 000300[1] 1787366079.470286 write 0x35eddb thru 0x35f152 (0x378 bytes) 000301[1] 1787366079.536767 write 0x84d98e thru 0x84f609 (0x1c7c bytes) 000302[1] 1787366079.544436 mapwrite 0x57e0ce thru 0x57f960 (0x1893 bytes) 000303[0] 1787366079.690820 mapread 0x585357 thru 0x586af4 (0x179e bytes) 000304[1] 1787366079.768522 trunc from 0x84f60a to 0x468c3f (0x3e69ca bytes) 000305[0] 1787366079.831287 mapwrite 0x981df9 thru 0x982802 (0xa0a bytes) 000307[0] 1787366079.998727 mapwrite 0x6e7ac8 thru 0x6e8c0c (0x1145 bytes) 000309[1] 1787366080.056791 mapwrite 0x6f365f thru 0x6f50b8 (0x1a5a bytes) 000310[1] 1787366080.208618 mapwrite 0x495d1a thru 0x4973d5 (0x16bc bytes) 000312[1] 1787366080.406873 mapread 0x117414 thru 0x117a65 (0x652 bytes) 000314[0] 1787366080.470081 read 0x3f1e23 thru 0x3f1e80 (0x5e bytes) 000315[0] 1787366080.490067 write 0x05e0a1 thru 0x05f335 (0x1295 bytes) 000316[0] 1787366080.529436 mapread 0x366ddf thru 0x36760d (0x82f bytes) 000317[1] 1787366080.563301 trunc from 0x982803 to 0x60eb7c (0x373c86 bytes) 000318[0] 1787366080.617975 mapread 0x3eea76 thru 0x3f02df (0x186a bytes) 000319[0] 1787366080.663068 write 0x1438a2 thru 0x14482e (0xf8d bytes) 000323[1] 1787366080.750941 trunc from 0x60eb7d to 0x75eddf (0x150263 bytes) 000324[1] 1787366080.814547 write 0x5e9570 thru 0x5eb47e (0x1f0f bytes) 000325[0] 1787366080.833190 write 0x45c482 thru 0x45e120 (0x1c9f bytes) 000327[1] 1787366080.887891 mapwrite 0x32e16b thru 0x32e4b9 (0x34f bytes) 000331[0] 1787366081.033399 trunc from 0x75ede0 to 0x0e2fc8 (0x67be17 bytes) 000332[1] 1787366081.110610 trunc from 0x0e2fc9 to 0x1c0965 (0xdd99d bytes) 000337[1] 1787366081.254218 trunc from 0x1c0966 to 0x1cecf8 (0xe393 bytes) 000338[0] 1787366081.309661 mapread 0x0e3807 thru 0x0e41b3 (0x9ad bytes) 000340[1] 1787366081.423900 read 0x0b506f thru 0x0b6ffc (0x1f8e bytes) 000341[1] 1787366081.537751 write 0x40e34d thru 0x40e600 (0x2b4 bytes) 000344[1] 1787366081.548861 mapwrite 0x740bb5 thru 0x74155c (0x9a8 bytes) 000347[1] 1787366081.945598 mapwrite 0x219763 thru 0x219ffa (0x898 bytes) 000349[1] 1787366082.008467 mapread 0x4c641c thru 0x4c8037 (0x1c1c bytes) 000350[0] 1787366082.009245 trunc from 0x74155d to 0x42af60 (0x3165fc bytes) 000356[0] 1787366082.207206 mapread 0x31f7aa thru 0x320d6f (0x15c6 bytes) 000357[1] 1787366082.235745 read 0x298ba0 thru 0x29a1c5 (0x1626 bytes) 000359[1] 1787366082.272478 mapwrite 0x84965b thru 0x84af84 (0x192a bytes) 000361[0] 1787366082.413354 mapwrite 0x81a7dc thru 0x81c33c (0x1b61 bytes) 000362[1] 1787366082.541945 read 0x7d2c22 thru 0x7d2cd5 (0xb4 bytes) 000363[0] 1787366082.611658 read 0x3371a9 thru 0x337fce (0xe26 bytes) 000364[0] 1787366082.668538 trunc from 0x84af85 to 0x5e980e (0x261776 bytes) 000365[1] 1787366082.777930 mapwrite 0x80e35c thru 0x80fe2a (0x1acf bytes) 000366[0] 1787366083.098037 trunc from 0x80fe2b to 0x43496a (0x3db4c0 bytes) 000367[1] 1787366083.156910 mapwrite 0x1736e3 thru 0x1751c5 (0x1ae3 bytes) 000368[0] 1787366083.353373 trunc from 0x43496b to 0x7d16bf (0x39cd55 bytes) 000372[0] 1787366083.458393 mapwrite 0x905d7f thru 0x9077e7 (0x1a69 bytes) 000374[1] 1787366083.584505 mapread 0x8c85b7 thru 0x8c8a32 (0x47c bytes) 000380[1] 1787366083.727923 read 0x63c669 thru 0x63c93e (0x2d6 bytes) 000381[0] 1787366083.820737 mapread 0x8ac10b thru 0x8acd6f (0xc65 bytes) 000382[0] 1787366083.850587 trunc from 0x9077e8 to 0x49a169 (0x46d67e bytes) 000386[0] 1787366084.082606 mapwrite 0x2aa254 thru 0x2aae6b (0xc18 bytes) 000389[1] 1787366084.226912 mapread 0x3e88d1 thru 0x3e961e (0xd4e bytes) 000392[0] 1787366084.344334 mapwrite 0x511bd4 thru 0x513506 (0x1933 bytes) 000393[0] 1787366084.455877 read 0x024b46 thru 0x024c75 (0x130 bytes) 000395[1] 1787366084.487270 read 0x2f841a thru 0x2f984b (0x1432 bytes) 000396[0] 1787366084.525501 write 0x1c25b6 thru 0x1c4290 (0x1cdb bytes) 000398[1] 1787366084.568166 mapread 0x0b5009 thru 0x0b64c8 (0x14c0 bytes) 000401[1] 1787366084.669081 read 0x236801 thru 0x2387d8 (0x1fd8 bytes) 000402[1] 1787366084.676160 mapread 0x4788ae thru 0x47a71a (0x1e6d bytes) 000403[1] 1787366084.720475 mapwrite 0x9affd8 thru 0x9b0b7f (0xba8 bytes) 000404[1] 1787366084.871519 trunc from 0x9b0b80 to 0x1468b5 (0x86a2ca bytes) 000405[0] 1787366084.896601 mapread 0x0f12e9 thru 0x0f2f46 (0x1c5e bytes) 000406[1] 1787366084.997341 mapread 0x0ac2f2 thru 0x0adcc8 (0x19d7 bytes) 000408[0] 1787366085.061640 read 0x007184 thru 0x0084ec (0x1369 bytes) 000414[0] 1787366085.198595 mapwrite 0x50c8dc thru 0x50e558 (0x1c7d bytes) 000418[0] 1787366085.615837 write 0x2d29c9 thru 0x2d41f2 (0x182a bytes) 000423[0] 1787366085.713473 trunc from 0x50e559 to 0x16bee0 (0x3a2678 bytes) 000428[1] 1787366085.862806 mapwrite 0x35ad2f thru 0x35b9b8 (0xc8a bytes) 000429[0] 1787366086.001144 write 0x73e1e3 thru 0x73fefd (0x1d1b bytes) 000430[0] 1787366086.050397 trunc from 0x73fefe to 0x09c911 (0x6a35ec bytes) 000439[0] 1787366086.153525 mapread 0x023f3a thru 0x0244b6 (0x57d bytes) 000440[1] 1787366086.220892 write 0x0663b2 thru 0x066e08 (0xa57 bytes) 000441[0] 1787366086.317217 write 0x457bb0 thru 0x458c24 (0x1075 bytes) 000444[0] 1787366086.416164 trunc from 0x458c25 to 0x57ca7f (0x123e5b bytes) 000448[0] 1787366086.461926 read 0x0b7044 thru 0x0b715a (0x117 bytes) 000449[1] 1787366086.474622 mapread 0x1bc991 thru 0x1be684 (0x1cf4 bytes) 000451[0] 1787366086.555962 trunc from 0x57ca80 to 0x46ccc5 (0x10fdba bytes) 000454[1] 1787366086.646224 write 0x7e93da thru 0x7e964a (0x271 bytes) 000456[0] 1787366086.689562 read 0x2edd6f thru 0x2edf2f (0x1c1 bytes) 000457[1] 1787366086.707074 write 0x34dc8d thru 0x34ddcf (0x143 bytes) 000462[1] 1787366086.771438 mapread 0x28f7d1 thru 0x290d2f (0x155f bytes) 000463[0] 1787366086.781008 read 0x7913e9 thru 0x792114 (0xd2c bytes) 000465[1] 1787366086.828680 read 0x07c5e8 thru 0x07cc60 (0x679 bytes) 000469[0] 1787366086.838918 mapwrite 0x6fab30 thru 0x6fc87e (0x1d4f bytes) 000471[0] 1787366086.940719 read 0x4e03ec thru 0x4e2344 (0x1f59 bytes) 000472[1] 1787366086.959886 trunc from 0x7e964b to 0x2d30f9 (0x516551 bytes) 000474[0] 1787366087.026050 mapread 0x2bab7b thru 0x2bbb56 (0xfdc bytes) 000476[0] 1787366087.079187 trunc from 0x2d30fa to 0x27814d (0x5afac bytes) 000477[0] 1787366087.117849 mapread 0x1ef662 thru 0x1ef9ff (0x39e bytes) 000479[0] 1787366087.128534 write 0x1aa372 thru 0x1ab780 (0x140f bytes) 000482[0] 1787366087.158970 mapread 0x1124c7 thru 0x1129dc (0x516 bytes) 000483[0] 1787366087.164896 trunc from 0x27814e to 0x14a00a (0x12e143 bytes) 000484[1] 1787366087.205579 mapwrite 0x076463 thru 0x077cd4 (0x1872 bytes) 000486[1] 1787366087.319431 mapwrite 0x5cfb42 thru 0x5cff35 (0x3f4 bytes) 000487[0] 1787366087.447836 read 0x57b296 thru 0x57b2f4 (0x5f bytes) 000488[0] 1787366087.481621 trunc from 0x5cff36 to 0x5c2989 (0xd5ac bytes) 000491[1] 1787366087.568420 write 0x02b1e4 thru 0x02bd13 (0xb30 bytes) 000492[1] 1787366087.600608 mapwrite 0x0910e2 thru 0x092931 (0x1850 bytes) 000495[0] 1787366087.831459 read 0x03b2e8 thru 0x03bdcf (0xae8 bytes) 000496[0] 1787366087.852907 read 0x1c6739 thru 0x1c7096 (0x95e bytes) 000497[0] 1787366087.909983 mapwrite 0x43353b thru 0x4346f7 (0x11bd bytes) 000499[0] 1787366087.978542 write 0x198651 thru 0x19961a (0xfca bytes) 000500[1] 1787366088.068152 read 0x12d710 thru 0x12e986 (0x1277 bytes) All operations completed A-OK! PASS 16b (30s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 22:34:54 (1787366094) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 22:34:59 (1787366099) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.582805 s, 14.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.544237 s, 15.4 MB/s ldlm.namespaces.MGC192.168.203.126@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9388c501a000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9388c8476000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9388c501a000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9388c501a000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5091 s, 8.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.15294 s, 25.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.3745 s, 31.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.14375 s, 14.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.16069 s, 11.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.60785 s, 29.1 MB/s PASS 16d (56s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 22:35:55 (1787366155) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.288713 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274938 s, 3.8 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 22:36:04 (1787366164) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 22:36:33 (1787366193) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 22:37:02 (1787366222) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 22:37:10 (1787366230) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 22:37:18 (1787366238) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.35789 s, 11.1 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.55381 s, 16.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.01717 s, 8.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.80432 s, 14.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.87066 s, 9.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.84306 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.61723 s, 7.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.33332 s, 19.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.71479 s, 15.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.59769 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.92371 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.26668 s, 11.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.38785 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.8465 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.01606 s, 8.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.64421 s, 7.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.01202 s, 6.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.28874 s, 6.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.53987 s, 5.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (44s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 22:38:02 (1787366282) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Aug 21 22:38:52 EDT 2026 + fsx 2 done: rc=0 Fri Aug 21 22:38:52 EDT 2026 + fsx 3 done: rc=0 Fri Aug 21 22:38:52 EDT 2026 + fsx 4 done: rc=0 Fri Aug 21 22:38:52 EDT 2026 + fsx 5 done: rc=0 Fri Aug 21 22:38:52 EDT 2026 PASS 16k (60s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 22:39:02 (1787366342) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 22:39:13 (1787366353) mmap test1: basic mmap operation (PASS, 0.1757s) mmap test2: MAP_PRIVATE not write back (PASS, 0.10338s) mmap test3: concurrent mmap ops on two nodes (PASS, 16.684s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 105.34s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.3057s) mmap test6: check mmap write/read content on two nodes (PASS, 0.44377s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.4664s) mmap test11: multi-thread shared mmap (PASS, 1.3015s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1731s) PASS 18 (136s) == sanityn test 19: test concurrent uncached read races ========================================================================= 22:41:29 (1787366489) oleg326-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 22:41:33 (1787366493) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 22:41:40 (1787366500) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 22:41:47 (1787366507) now is 1787366509 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 new atime is 1787366570 PASS 23 (69s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 22:42:57 (1787366577) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 22:43:04 (1787366584) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 22:43:11 (1787366591) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 22:43:19 (1787366599) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 22:43:22 (1787366602) utime: good mknod times 1787366603 <= 1787366604 <= 1787366604 for /mnt/lustre/f26a utime: good mknod times 1787366603 <= 1787366604 <= 1787366604 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 22:43:30 (1787366610) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 22:43:40 (1787366620) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-21 22:43:42.000000000 -0400 Birth: 2026-08-21 22:43:43.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502395 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-21 22:43:42.000000000 -0400 Birth: 2026-08-21 22:43:43.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 22:43:50 (1787366630) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 22:43:59 (1787366639) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 22:44:07 (1787366647) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.126@tcp:/lustre 7542784 38912 7497728 1% /mnt/lustre2 PASS 31b (24s) == sanityn test 31r: open-rename(replace) race =========== 22:44:31 (1787366671) -rw-r--r-- 1 root root 0 Aug 21 22:44 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 22:44:42 (1787366682) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43157 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 22:44:49 (1787366689) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 22:44:57 (1787366697) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 22:45:00 (1787366700) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53704 s, 34.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.32665 s, 39.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.70937 s, 30.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.9971 s, 26.3 MB/s contention_seconds 4, contention_events 16 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.81657 s, 28.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.05871 s, 25.5 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.09395 s, 47.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.13487 s, 46.2 MB/s PASS 32c (27s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 22:45:30 (1787366730) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 22:45:34 (1787366734) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 22:45:37 (1787366737) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 22:45:41 (1787366741) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 22:45:44 (1787366744) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9388c501a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9388c501a000.ost_server_uuid in FULL state after 0 sec oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9388c501a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9388c501a000.ost_server_uuid in IDLE state after 0 sec oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9388c501a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9388c501a000.ost_server_uuid in FULL state after 0 sec oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9388c501a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9388c501a000.ost_server_uuid in IDLE state after 0 sec oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9388c501a000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9388c501a000.ost_server_uuid in FULL state after 0 sec oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9388c501a000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9388c501a000.ost_server_uuid in IDLE state after 0 sec PASS 34 (133s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 22:47:57 (1787366877) fail_loc=0x80000317 Race attempt 0 Wait for 50462 50514 for 60 sec... - open/close 694 (time 1787366893.79 total 10.01 last 69.30) - open/close 1311 (time 1787366903.79 total 20.01 last 61.70) - open/close 1984 (time 1787366913.82 total 30.04 last 67.11) - open/close 2705 (time 1787366923.82 total 40.05 last 72.06) - open/close 3360 (time 1787366933.84 total 50.07 last 65.38) total: 4000 open/close in 59.32 seconds: 67.43 ops/second /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50462 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.126@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre 192.168.203.126@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre2 PASS 35 (72s) == sanityn test 36: handle ESTALE/open-unlink correctly == 22:49:09 (1787366949) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg326-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.74296 s, 30.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7499776) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.77176 s, 29.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7495680) after_dd(7448576) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2062 s, 43.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7499776) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.64245 s, 31.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7446528) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.65554 s, 31.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7499776) after_dd(7444480) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.89889 s, 27.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7499776) after_dd(7448576) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.96108 s, 26.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7499776) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.95998 s, 26.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(7) *** before(7494656) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.5352 s, 34.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.72736 s, 30.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37359 s, 38.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7441408) after(7498752) PASS 36 (343s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 22:54:52 (1787367292) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 - create 1257 (time 1787367307.24 total 10.00 last 125.65) - create 2495 (time 1787367317.24 total 20.01 last 123.76) - create 3740 (time 1787367327.25 total 30.01 last 124.43) - create 5001 (time 1787367337.26 total 40.02 last 126.01) - create 6285 (time 1787367347.27 total 50.04 last 128.18) - create 7606 (time 1787367357.28 total 60.04 last 132.00) - create 8902 (time 1787367367.28 total 70.05 last 129.58) total: 10000 create in 78.73 seconds: 127.01 ops/second PASS 37 (91s) == sanityn test 39a: file mtime does not change after rename ========================================================== 22:56:23 (1787367383) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 22:56:31 (1787367391) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 22:56:40 (1787367400) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 22:56:49 (1787367409) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 22:57:00 (1787367420) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (22s) == sanityn test 40c: pdirops: link and others ======================================================================== 22:57:23 (1787367443) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 22:57:41 (1787367461) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (19s) == sanityn test 40e: pdirops: rename and others ======================================================================== 22:58:00 (1787367480) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 22:58:18 (1787367498) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (15s) == sanityn test 41b: pdirops: create vs create ======================================================================== 22:58:33 (1787367513) open: File exists Conflict PASS 41b (16s) == sanityn test 41c: pdirops: create vs link ======================================================================== 22:58:49 (1787367529) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 22:59:03 (1787367543) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 22:59:19 (1787367559) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 22:59:33 (1787367573) Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 22:59:46 (1787367586) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 22:59:58 (1787367598) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 23:00:11 (1787367611) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1100s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 23:18:31 (1787368711) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 23:18:44 (1787368724) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 23:18:57 (1787368737) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 23:19:10 (1787368750) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 23:19:23 (1787368763) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 23:19:36 (1787368776) Conflict PASS 42f (12s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 23:19:48 (1787368788) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 23:20:02 (1787368802) Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 23:20:17 (1787368817) PASS 43a (80s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 23:21:38 (1787368898) Conflict PASS 43b (13s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 23:21:50 (1787368910) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 23:22:04 (1787368924) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 23:22:18 (1787368938) Conflict PASS 43e (12s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 23:22:30 (1787368950) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 23:22:45 (1787368965) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (17s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 23:23:01 (1787368981) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 23:23:16 (1787368996) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 23:23:20 (1787369000) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (132s) == sanityn test 43k: unlink vs create ==================== 23:25:32 (1787369132) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1147s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 23:44:39 (1787370279) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Aug 21 23:44:48 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 23:44:52 (1787370292) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 23:45:06 (1787370306) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (13s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 23:45:20 (1787370320) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 23:45:34 (1787370334) Conflict PASS 44e (13s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 23:45:47 (1787370347) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 23:46:01 (1787370361) Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 23:46:15 (1787370375) Conflict PASS 44h (15s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 23:46:30 (1787370390) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 23:46:35 (1787370395) PASS 45a (138s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 23:48:52 (1787370532) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 23:49:05 (1787370545) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 23:49:18 (1787370558) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 23:49:32 (1787370572) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 23:49:47 (1787370587) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 23:50:01 (1787370601) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (12s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 23:50:13 (1787370613) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 23:50:26 (1787370626) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 23:50:29 (1787370629) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1219s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 00:10:47 (1787371847) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (15s) == sanityn test 46b: pdirops: link vs create ======================================================================== 00:11:02 (1787371862) open: File exists Conflict PASS 46b (14s) == sanityn test 46c: pdirops: link vs link ======================================================================== 00:11:16 (1787371876) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (16s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 00:11:32 (1787371892) Conflict PASS 46d (16s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 00:11:48 (1787371908) Conflict PASS 46e (16s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 00:12:04 (1787371924) Conflict PASS 46f (14s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 00:12:19 (1787371939) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 00:12:33 (1787371953) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 00:12:47 (1787371967) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 00:12:50 (1787371970) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 00:12:54 (1787371974) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 00:12:58 (1787371978) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 00:13:02 (1787371982) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 00:13:06 (1787371986) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 00:13:10 (1787371990) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 00:13:13 (1787371993) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 00:13:16 (1787371996) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.185426 s, 55.2 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 00:13:29 (1787372009) 0+1 records in 0+1 records out 158 bytes copied, 0.0218788 s, 7.2 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218856) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 00:13:39 (1787372019) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0133216 s, 76.9 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094995 s, 11.0 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 00:14:00 (1787372040) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0178009 s, 57.5 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 00:14:13 (1787372053) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385074 s, 27.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 00:14:21 (1787372061) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (11s) == sanityn test 54: rename locking ======================= 00:14:32 (1787372072) llite.lustre-ffff9388c501a000.xattr_cache=1 llite.lustre-ffff9388c8476000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505012 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 00:14:34.000000000 -0400 Modify: 2026-08-22 00:14:34.000000000 -0400 Change: 2026-08-22 00:14:34.000000000 -0400 Birth: 2026-08-22 00:14:35.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 00:14:40.000000000 -0400 Modify: 2026-08-22 00:14:42.000000000 -0400 Change: 2026-08-22 00:14:42.000000000 -0400 Birth: 2026-08-22 00:14:41.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 00:14:47.000000000 -0400 Modify: 2026-08-22 00:14:47.000000000 -0400 Change: 2026-08-22 00:14:47.000000000 -0400 Birth: 2026-08-22 00:14:48.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 00:14:47.000000000 -0400 Modify: 2026-08-22 00:14:48.000000000 -0400 Change: 2026-08-22 00:14:48.000000000 -0400 Birth: 2026-08-22 00:14:48.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 00:14:53.000000000 -0400 Modify: 2026-08-22 00:14:54.000000000 -0400 Change: 2026-08-22 00:14:54.000000000 -0400 Birth: 2026-08-22 00:14:54.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 00:14:53.000000000 -0400 Modify: 2026-08-22 00:14:54.000000000 -0400 Change: 2026-08-22 00:14:54.000000000 -0400 Birth: 2026-08-22 00:14:54.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (33s) == sanityn test 55a: rename vs unlink target dir ========= 00:15:05 (1787372105) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 00:15:19 (1787372119) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 00:15:33 (1787372133) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (18s) == sanityn test 55d: rename file vs link ================= 00:15:52 (1787372152) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 00:16:07 (1787372167) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 00:16:10 (1787372170) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 00:16:30 (1787372190) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (24s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 00:16:54 (1787372214) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1787372216 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.565177 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (116s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 00:18:50 (1787372330) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1787372335 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.946312 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (142s) == sanityn test 60: Verify data_version behaviour ======== 00:21:12 (1787372472) PASS 60 (10s) == sanityn test 70a: cd directory && rm directory ======== 00:21:22 (1787372482) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 00:21:31 (1787372491) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 00:21:42 (1787372502) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0134601 s, 4.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 00:21:45 (1787372505) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0246468 s, 2.7 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 00:21:49 (1787372509) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 00:21:52 (1787372512) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0246304 s, 2.7 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 00:21:56 (1787372516) llite.lustre-ffff9388c501a000.xattr_cache=1 llite.lustre-ffff9388c8476000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (9s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 00:22:04 (1787372524) llite.lustre-ffff9388c501a000.xattr_cache=1 llite.lustre-ffff9388c8476000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (8s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 00:22:12 (1787372532) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0137005 s, 74.7 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0202077 s, 50.7 kB/s 231672: get lock2 231673: get lock1 231672: done 231672: get lock1 231673: done 231673: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231673: cannot get lock2: Resource deadlock avoided 231673: exit rc=0 231672: done 231672: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 00:22:23 (1787372543) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0909228 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303902 s, 34.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232253 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 00:22:35 (1787372555) opening files: ........ closing files: ........ PASS 76 (74s) == sanityn test 77a: check FIFO NRS policy =============== 00:23:49 (1787372629) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.444008 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610399 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100945 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104451 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111387 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129608 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168847 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16111 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810332 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516717 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193985 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183456 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233729 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258951 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320597 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373213 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384611 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101278 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146152 s, 71.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248364 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949285 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435822 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753986 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499379 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00579326 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570492 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417108 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487129 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00284915 s, 368 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021367 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569837 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579913 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00906826 s, 116 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 00:23:58 (1787372638) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.494095 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150329 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882415 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763736 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105058 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246345 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1273 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241004 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223242 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253969 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300646 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278076 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240802 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334471 s, 3.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303247 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318533 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412916 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594672 s, 176 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783564 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135509 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635868 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168308 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946495 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099698 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991275 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403002 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115323 s, 90.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150104 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00560485 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812945 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124854 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15429 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343932 s, 30.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.482527 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137306 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120584 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11195 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195852 s, 5.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201054 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240415 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17594 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102877 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103402 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260694 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211115 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1991 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175807 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219393 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256279 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293329 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10265 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140259 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107395 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117065 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167199 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138761 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139738 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146385 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145509 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148844 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208618 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113381 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15091 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153155 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210663 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16981 s, 6.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (15s) == sanityn test 77c: check ORR NRS policy ================ 00:24:13 (1787372653) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.470804 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622489 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061846 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957491 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779846 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109471 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11225 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131141 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108927 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134823 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177353 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182745 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173918 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20434 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24156 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217257 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218961 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600994 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125408 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00699003 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129256 s, 81.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147484 s, 71.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520607 s, 201 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107637 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129676 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01189 s, 88.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802518 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12817 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816033 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135759 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871565 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300331 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979445 s, 10.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.644428 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617382 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709415 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113087 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232914 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220751 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232953 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245027 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216985 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24979 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254047 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263834 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245239 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275234 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266461 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286865 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281593 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387612 s, 27.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714891 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347671 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527988 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489122 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064027 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101502 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946401 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593807 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744948 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960182 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734724 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595165 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528886 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540956 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785713 s, 13.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.523263 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166512 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159401 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204543 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203855 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205133 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2792 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255387 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19438 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300915 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222726 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370466 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399576 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390565 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340805 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367301 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496287 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00655456 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122126 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158532 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162038 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00703972 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00957044 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102732 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986141 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018516 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109087 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132883 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654981 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833932 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633725 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555453 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660107 s, 15.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (16s) == sanityn test 77d: check TRR nrs policy ================ 00:24:29 (1787372669) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.392426 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290046 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596053 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525357 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740674 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152317 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116777 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127008 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106544 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156579 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157767 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241927 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235871 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239045 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285335 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298566 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309859 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257526 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500155 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931869 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106184 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00257425 s, 407 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00680071 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00264393 s, 397 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387971 s, 270 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131954 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183659 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835533 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0053082 s, 198 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091082 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101436 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454959 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253455 s, 41.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.696373 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925674 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615726 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113947 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126339 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232015 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242383 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286697 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357608 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26223 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368422 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362962 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286687 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304096 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305268 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411079 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463963 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202815 s, 51.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682073 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586363 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769199 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884113 s, 11.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806793 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139851 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100083 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10434 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110805 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114486 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114555 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911543 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110102 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124878 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132444 s, 7.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.642037 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419921 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135827 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10845 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138206 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113457 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118484 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197796 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269096 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317687 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313001 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26574 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363543 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380762 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356453 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273645 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399457 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032026 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639552 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373021 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605045 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044385 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475845 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062531 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689493 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806345 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854778 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108038 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111351 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974209 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160616 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966232 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113542 s, 9.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (17s) == sanityn test 77e: check TBF NID nrs policy ============ 00:24:47 (1787372687) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.26@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.436877 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430775 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658631 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793406 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714854 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859435 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105556 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739008 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788207 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806762 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156089 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151349 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19072 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242194 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21666 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190319 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255913 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806597 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806034 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668083 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810057 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709304 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942055 s, 11.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890671 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117786 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10285 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108059 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108913 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100669 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114089 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114796 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11787 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107888 s, 9.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.468501 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816796 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911666 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950501 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693679 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130897 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898092 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11437 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125681 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156972 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213267 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209281 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273678 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290992 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26431 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317345 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935241 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221031 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0098768 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566723 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307479 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934581 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358857 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392844 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00471976 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00805263 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520373 s, 202 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625872 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908556 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826686 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00531158 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781581 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402328 s, 26.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.57016 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553692 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745175 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865509 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151154 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183443 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19325 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162533 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224477 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236692 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2156 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184302 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24889 s, 4.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219135 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20047 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250209 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230993 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00806341 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954432 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815038 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816562 s, 12.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102936 s, 10.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373353 s, 28.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870289 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404747 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193879 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106787 s, 98.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00617263 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137165 s, 76.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985574 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00771077 s, 136 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00812845 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00580859 s, 181 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.385446 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569216 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615046 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086241 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113814 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196044 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193747 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142075 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21955 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173816 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18657 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177515 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285056 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237413 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254854 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194421 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302581 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452886 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447844 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391323 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505321 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07448 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617431 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533118 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732024 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613521 s, 17.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502643 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583422 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542282 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706196 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537941 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550829 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857431 s, 12.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (21s) == sanityn test 77f: check TBF JobID nrs policy ========== 00:25:08 (1787372708) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.336841 s, 49.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494171 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547196 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900393 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938508 s, 11.2 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181863 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17454 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150764 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2158 s, 4.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176582 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22867 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216958 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213732 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17053 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177383 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161143 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165168 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00708918 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778353 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010318 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686599 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477085 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00856862 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00871491 s, 120 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334515 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588122 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158477 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104936 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741076 s, 14.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245622 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141475 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794193 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106554 s, 98.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.403716 s, 41.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10232 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10395 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114234 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101801 s, 10.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841506 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887528 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110563 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133368 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190656 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179964 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165115 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150147 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108427 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168748 s, 6.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192142 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188517 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444976 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309762 s, 33.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365162 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047496 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469617 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201338 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335773 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213235 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391991 s, 26.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364497 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404783 s, 25.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356736 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457755 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396557 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397012 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463159 s, 22.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.407644 s, 41.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439477 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692449 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888349 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115064 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114773 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11757 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160639 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180999 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187828 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188446 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172285 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227588 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182528 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230326 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193116 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173149 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018228 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016125 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130111 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267965 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552795 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255053 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206878 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269134 s, 39.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562918 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311483 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05756 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029543 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371508 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368152 s, 28.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389242 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373369 s, 28.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.44033 s, 38.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117982 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752793 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110103 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894022 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232221 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220334 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24188 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228004 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24001 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287131 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33783 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356209 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204708 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371589 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198705 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373632 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556022 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368143 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578045 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019606 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961036 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00852269 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581504 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360448 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675887 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722504 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446119 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483313 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539383 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561555 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597727 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621419 s, 16.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (18s) == sanityn test 77g: Change TBF type directly ============ 00:25:26 (1787372726) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 00:25:37 (1787372737) oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg326-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg326-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg326-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg326-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 PASS 77h (11s) == sanityn test 77i: Change rank of TBF rule ============= 00:25:48 (1787372748) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (23s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 00:26:11 (1787372771) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.607196 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503436 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541341 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766581 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165093 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205983 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215886 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265995 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273355 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227359 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279598 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26267 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253762 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252005 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261516 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224717 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263345 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230658 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22805 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290743 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27714 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25834 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230225 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249754 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263038 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264077 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274485 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280707 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297106 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303711 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302906 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308545 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353686 s, 3.0 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.11982 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.571 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (64s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 00:27:16 (1787372836) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg326-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg326-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg326-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg326-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31137 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952145 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129574 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101687 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140063 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496104 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136611 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107289 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169064 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120941 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473248 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183924 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21795 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173665 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225959 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21766 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223725 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697203 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736988 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716146 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857964 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929899 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960719 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094583 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104019 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106931 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893086 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119009 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099063 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11065 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112853 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10692 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113583 s, 9.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9937 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9585 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg326-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg326-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg326-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg326-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28501 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112756 s, 9.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107708 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157008 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169951 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174147 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166536 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205179 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244808 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264809 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246858 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291142 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270174 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263526 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244945 s, 4.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306277 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266767 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505787 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683988 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714066 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723649 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050964 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839137 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630126 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694945 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630204 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599667 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920181 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794116 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792625 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752472 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798441 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086306 s, 12.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9412 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9226 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (155s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 00:29:51 (1787372991) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg326-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg326-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg326-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg326-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27449 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110775 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115053 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128668 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132491 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138071 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312825 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288741 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340777 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328862 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329702 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324923 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330533 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320619 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350886 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258072 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307302 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044664 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719499 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055802 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701616 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878172 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640314 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974801 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098989 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106458 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110576 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954886 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918298 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124508 s, 8.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989359 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974518 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101374 s, 10.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0794 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9815 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg326-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg326-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg326-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg326-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3038 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409533 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421159 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867201 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977646 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239089 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233495 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265821 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222305 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208657 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275374 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216914 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235202 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29024 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293411 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281699 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357473 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101837 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102527 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116442 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974856 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130217 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109725 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12391 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143476 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124203 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11554 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134904 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10241 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136938 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123307 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117503 s, 8.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0124 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9511 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (153s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 00:32:24 (1787373144) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.607034 s, 27.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694845 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104383 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231092 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206687 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215521 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300978 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303928 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352303 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341544 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274938 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381653 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34226 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372792 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354288 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404481 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465948 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730837 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447212 s, 23.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102967 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625525 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635273 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675462 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109844 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744635 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758378 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764017 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784393 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133266 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089038 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914758 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961212 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104326 s, 10.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.33633 s, 12.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92136 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.26@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.26@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.532878 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853194 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854207 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104891 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155806 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19086 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230877 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251468 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279379 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260906 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250224 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248599 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224219 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269642 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247011 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255928 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318314 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719545 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612833 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450961 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462807 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556832 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808079 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540063 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716965 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551146 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539316 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679514 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063019 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722118 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717509 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714598 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758924 s, 13.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.88493 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97428 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.26@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.562974 s, 29.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482581 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505681 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12597 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127524 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201956 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173518 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222543 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210539 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203064 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214218 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221218 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21553 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233078 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220047 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216928 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243818 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270586 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238119 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427933 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429825 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384892 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470614 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551282 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489797 s, 21.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563301 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601533 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451502 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823868 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646508 s, 16.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093497 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068234 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714104 s, 14.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.92234 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18693 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.867831 s, 19.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539611 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117875 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111926 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145377 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264128 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301098 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278449 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2476 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256235 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360317 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339628 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279472 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333359 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25497 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269564 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353103 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833396 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436852 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505591 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550759 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594493 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588357 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425561 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646247 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525435 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687024 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776069 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690552 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856255 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864238 s, 12.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895843 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106493 s, 9.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.635 s, 9.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9408 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.20301 s, 11.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92104 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.03724 s, 16.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903293 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867827 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24022 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238811 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294054 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271435 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259535 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248783 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268477 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269872 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293117 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320467 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301595 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368834 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328728 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363475 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602588 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328769 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471035 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00643843 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329511 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726267 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735328 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159102 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408131 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024025 s, 436 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101987 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825683 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215196 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013762 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00738922 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131727 s, 79.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7923 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6102 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.697475 s, 24.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483638 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525954 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397721 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476993 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957487 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073535 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114246 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874771 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120396 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179924 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211409 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17571 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220099 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213733 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24322 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250836 s, 4.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522186 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418607 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708147 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643706 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081027 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572075 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848122 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880802 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119391 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761129 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768758 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840895 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891065 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113363 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09626 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104359 s, 10.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.00604 s, 13.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91698 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.59979 s, 28.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379387 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320225 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338639 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143416 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135879 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170376 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162891 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146728 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175063 s, 6.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187957 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184636 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173401 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213779 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174382 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238442 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233147 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00973236 s, 108 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473966 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108345 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930587 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746033 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976988 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848922 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00971978 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018053 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105025 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121992 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114606 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110844 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118063 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00664507 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00924944 s, 113 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0547 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92904 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1786 s, 9.4 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0251 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (433s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 00:39:37 (1787373577) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.808407 s, 20.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417438 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733491 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831901 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050723 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494378 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153187 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215236 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248974 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16931 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224405 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236273 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350494 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329162 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358789 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289347 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327172 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064726 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10059 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00560793 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135183 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450287 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707326 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00761096 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338159 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00730497 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00517879 s, 202 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239406 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201979 s, 51.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120241 s, 87.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253596 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00667856 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202487 s, 51.8 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.95603 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92428 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (51s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 00:40:28 (1787373628) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.26@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.26@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5327 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898774 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840707 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220374 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222664 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22492 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270017 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287914 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246935 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288089 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282843 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308268 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330689 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261671 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316329 s, 3.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303882 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354697 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569559 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877511 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647389 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071744 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652703 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043702 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797255 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919823 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723826 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932395 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557722 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830678 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576476 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598367 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602219 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582117 s, 18.0 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.13795 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9199 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (49s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 00:41:17 (1787373677) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.26@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.523833 s, 32.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147377 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134644 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153493 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120258 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193046 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161287 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154905 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193243 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174471 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205584 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134543 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168352 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804836 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196084 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170895 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190968 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271603 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045028 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232943 s, 45.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288762 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297762 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295017 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625395 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410543 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609704 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456132 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450546 s, 23.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554928 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579203 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592862 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604691 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628163 s, 16.7 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.18364 s, 14.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06882 s, 20.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (42s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 00:41:59 (1787373719) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.651296 s, 25.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078918 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087105 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056759 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541449 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999543 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927666 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986954 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204534 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11334 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185927 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179603 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194302 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123838 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088557 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311505 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192365 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634122 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342286 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383567 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486906 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586768 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560456 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519072 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658837 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959299 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954322 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501001 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571667 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629406 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702915 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10916 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758078 s, 13.8 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7345 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92212 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.16168 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95119 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (90s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 00:43:29 (1787373809) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.021 s, 16.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057673 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121051 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119125 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101563 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112059 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132909 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137775 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940132 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720055 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228308 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219239 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208377 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239067 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278317 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156697 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282817 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440101 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219295 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040278 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438286 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505766 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439281 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841211 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694854 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723242 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852268 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731116 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081243 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707209 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778963 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892716 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907607 s, 11.6 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8857 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9694 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (75s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 00:44:44 (1787373884) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.633336 s, 26.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672168 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778972 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10447 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265148 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189542 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25189 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198568 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283657 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294494 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242869 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311785 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292995 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308998 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340349 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371933 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348025 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296133 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319909 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177295 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890139 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385359 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427699 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00643943 s, 163 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00771852 s, 136 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00525529 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984973 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08712 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0065166 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111837 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115025 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111522 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195477 s, 53.6 MB/sldlm.namespaces.lustre-OST0000-osc-ffff9388c8476000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9388c8476000.lock_unused_count=1 removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.25402 s, 11.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94953 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg326-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.643869 s, 26.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267536 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851385 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106599 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234013 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206362 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237871 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238269 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245171 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20913 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233195 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313205 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279345 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273566 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307179 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352806 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32814 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651264 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596724 s, 17.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605494 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609474 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648444 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985072 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654128 s, 16.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696334 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785186 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077891 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798311 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816669 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941638 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899724 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921394 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107307 s, 9.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2619 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95907 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.405 s, 10.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90915 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (143s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 00:47:07 (1787374027) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg326-client.virtnet /mnt/lustre (opts:) Stopping client oleg326-client.virtnet /mnt/lustre opts: Stopping clients: oleg326-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg326-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg326-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg326-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.26,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg326-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.26,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg326-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.26,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/68418/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.26,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg326-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing load_modules_local oleg326-server: Loading modules from /home/green/git/lustre-release/lustre oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg326-server: detected 4 online CPUs by sysfs oleg326-server: MODOPTS_LIBCFS= oleg326-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre Starting client oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre Started clients oleg326-client.virtnet: 192.168.203.126@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Starting client oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 Started clients oleg326-client.virtnet: 192.168.203.126@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9388c3401800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9388faa7b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9388c3401800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9388faa7b000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.543931 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081527 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107187 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135473 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189037 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177861 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189751 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112669 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148283 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161329 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180832 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223119 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253454 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316746 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316583 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313883 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388faa7b000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9388faa7b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218955 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445857 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461982 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461587 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475895 s, 2.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.464171 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458931 s, 2.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.504966 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461271 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45155 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460603 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.512761 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468564 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462641 s, 2.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.469165 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482776 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483085 s, 2.2 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.49164 s, 12.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13535 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.656151 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598484 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695965 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153784 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230577 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22614 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257236 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226607 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268376 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277174 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227279 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228231 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288345 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282385 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27429 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274928 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297283 s, 3.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9388faa7b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451515 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875759 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992832 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772845 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831137 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843645 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523196 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468296 s, 22.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576228 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856624 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734439 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929678 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764596 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709885 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806946 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805916 s, 13.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.48067 s, 14.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95156 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (211s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 00:50:38 (1787374238) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg326-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 00:50:55 (1787374255) On MGS 192.168.203.126, active = nodemap.active=1 On MGS 192.168.203.126, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.781234 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761351 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104626 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11861 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484327 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188827 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218296 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196155 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906648 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102257 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349866 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195191 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342837 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388faa7b000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9388faa7b000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366414 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357159 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351638 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437759 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372208 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3446 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27944 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345665 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394448 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395539 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417567 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362167 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390044 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439903 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376306 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388287 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374805 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348658 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3733 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456226 s, 2.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.60745 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3446 s, 19.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.126, TBF.id = On MGS 192.168.203.126, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.647376 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524514 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599779 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157975 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210074 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181773 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228281 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211744 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213544 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206865 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169556 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215605 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234254 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246839 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181374 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196282 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230562 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388faa7b000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9388faa7b000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579214 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356284 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515456 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528313 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033358 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513954 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544686 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438835 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659943 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471045 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643095 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734371 s, 14.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602341 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775374 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950973 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976884 s, 10.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.09221 s, 13.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.10114 s, 33.8 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg326-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg326-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 On MGS 192.168.203.126, active = nodemap.active=1 On MGS 192.168.203.126, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.535236 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448291 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989835 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266682 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245093 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237928 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214347 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234034 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193791 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259599 s, 4.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256625 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311922 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383673 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32269 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302504 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385118 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328341 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388faa7b000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9388faa7b000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17476 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185018 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185262 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186311 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203074 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193347 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196851 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202156 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198075 s, 5.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209991 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259649 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229123 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233985 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240861 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241192 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260193 s, 4.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9388faa7b000.lock_unused_count=1 removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.80191 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92615 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.126, TBF.id = On MGS 192.168.203.126, active = nodemap.active=0 PASS 77kj (184s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 00:53:59 (1787374439) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 00:54:11 (1787374451) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg326-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 4.0721 s, 258 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg326-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.065 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (57s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 00:55:09 (1787374509) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.561198 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926541 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996428 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669224 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117893 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141271 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138797 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316029 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214932 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37708 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248208 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331175 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377071 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359164 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254527 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408372 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356988 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957983 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661074 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672603 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053147 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590897 s, 17.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9388faa7b000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9388faa7b000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572005 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664202 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060049 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650857 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734196 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074523 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814301 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849858 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836748 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128172 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138562 s, 7.6 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg326-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg326-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.01586 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12897 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.580574 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07226 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718001 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804653 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920329 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074938 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929844 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145815 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726474 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853644 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113662 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140961 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136426 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133663 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131401 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165641 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15183 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647309 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918847 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758483 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897894 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706975 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00901463 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630213 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692747 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474757 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521566 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00789143 s, 133 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154441 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929518 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698461 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743173 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923738 s, 11.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg326-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg326-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83174 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27409 s, 19.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (91s) == sanityn test 77o: Changing rank should not panic ====== 00:56:40 (1787374600) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (11s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 00:56:50 (1787374610) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (118s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 00:58:48 (1787374728) mds.MDS.mdt.nrs_policies=tbf oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg326-client: oleg326-server: ssh exited with exit code 36 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 oleg326-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg326-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg326-client: oleg326-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (35s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 00:59:23 (1787374763) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273845 kill md_thread xargs PASS 77r (61s) == sanityn test 77s: Check TBF LRU shrinker ============== 01:00:24 (1787374824) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (70s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 01:01:34 (1787374894) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 01:01:45 (1787374905) File: /mnt/lustre/d79.sanityn Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935727 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 01:01:47.000000000 -0400 Modify: 2026-08-22 01:01:47.000000000 -0400 Change: 2026-08-22 01:01:47.000000000 -0400 Birth: 2026-08-22 01:01:48.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (20s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 01:02:04 (1787374924) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 01:02:08 (1787374928) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 01:02:11 (1787374931) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 01:02:14 (1787374934) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 01:02:18 (1787374938) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (2s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 01:02:21 (1787374941) - open/close 767 (time 1787374956.17 total 10.01 last 76.60) - open/close 1456 (time 1787374966.18 total 20.02 last 68.84) - open/close 2169 (time 1787374976.20 total 30.04 last 71.19) total: 2500 open/close in 34.99 seconds: 71.46 ops/second snapshot_time 1787375073.389406232 secs.nsecs start_time 1787374944.820782493 secs.nsecs elapsed_time 128.568623739 secs.nsecs open 2505 samples [usecs] 144 12818 3820773 9451222565 close 2504 samples [usecs] 186 6782 925490 779157166 mknod 2501 samples [usecs] 566 12535 3334550 7264209182 unlink 2500 samples [usecs] 739 63397 6331483 57443257183 mkdir 2 samples [usecs] 1064 9323 10387 88050425 rmdir 2 samples [usecs] 10296 12701 22997 267323017 rename 2500 samples [usecs] 482 48444 18229643 257209682321 getattr 3762 samples [usecs] 7 6694 266498 348725184 parallel_rename_file 2500 samples [usecs] 482 48444 18229643 257209682321 crossdir_rename 2500 samples [usecs] 482 48444 18229643 257209682321 readpage 26 samples [usecs] 4307 46974 526374 13583041974 PASS 81d (136s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 01:04:37 (1787375077) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7635 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7635 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 01:04:44 (1787375084) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 01:04:47 (1787375087) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 01:05:00 (1787375100) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1787375102 Finishing test test1 at 1787375106 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 01:05:12 (1787375112) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 01:05:15 (1787375115) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 01:05:18 (1787375118) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 01:05:22 (1787375122) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5859 0x16e3 0x240000400 1 5821 0x16bd 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5860 0x16e4 0x240000400 1 5822 0x16be 0x280000400 rc1=0 and rc2=0 PASS 93 (18s) == sanityn test 94: signal vs CP callback race =========== 01:05:40 (1787375140) osc.lustre-OST0000-osc-ffff9388c3401800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9388faa7b000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9388c3401800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9388faa7b000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.325438 s, 25.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.08926 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290928 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.069915 s, 58.6 kB/s osc.lustre-OST0000-osc-ffff9388c3401800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9388faa7b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9388c3401800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9388faa7b000.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 01:05:55 (1787375155) llite.lustre-ffff9388c3401800.fast_read=0 llite.lustre-ffff9388faa7b000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104627 s, 20.0 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9388c3401800.fast_read=1 llite.lustre-ffff9388faa7b000.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 01:06:16 (1787375176) llite.lustre-ffff9388c3401800.fast_read=0 llite.lustre-ffff9388faa7b000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0231696 s, 530 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9388c3401800.fast_read=1 llite.lustre-ffff9388faa7b000.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 01:06:30 (1787375190) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (5s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 01:06:34 (1787375194) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0123793 s, 331 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 01:06:42 (1787375202) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154361 s, 13.6 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 01:06:51 (1787375211) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172232 s, 12.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 01:06:58 (1787375218) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0121159 s, 990 kB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 01:07:05 (1787375225) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00866128 s, 473 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 01:07:13 (1787375233) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0136082 s, 301 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 01:07:20 (1787375240) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00403491 s, 1.0 MB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 01:07:28 (1787375248) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 90 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf90:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (7s) == sanityn test 103: Test size correctness with lockahead ========================================================== 01:07:36 (1787375256) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1787375259 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1787375262 Finishing test test23 at 1787375262 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 01:07:49 (1787375269) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 01:07:53 (1787375273) fail_loc=0x416 PASS 105 (22s) == sanityn test 106a: Verify the btime via statx() ======= 01:08:15 (1787375295) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 01:08:18 (1787375298) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0257671 s, 39.7 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 01:08:19.000000000 -0400 Modify: 2026-08-22 01:08:20.000000000 -0400 Change: 2026-08-22 01:08:20.000000000 -0400 Birth: 2026-08-22 01:08:21.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938263 -rw-r--r-- 1 0 0 1787375301 1787375299 1787375300 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 01:08:19.000000000 -0400 Modify: 2026-08-22 01:08:20.000000000 -0400 Change: 2026-08-22 01:08:20.000000000 -0400 Birth: 2026-08-22 01:08:21.000000000 -0400 1787375300 1024 11 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 01:08:25 (1787375305) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 01:08:32 (1787375312) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.217061 s, 48.3 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5869 0x16ed 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 01:08:42 (1787375322) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.437907 s, 23.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5829 0x16c5 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (15s) == sanityn test 108a: lseek: parallel updates ============ 01:08:57 (1787375337) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0114144 s, 876 kB/s llite.lustre-ffff9388c3401800.hybrid_io=0 llite.lustre-ffff9388faa7b000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.77887 s, 1.8 MB/s PASS 108a (14s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 01:09:11 (1787375351) 192.168.203.126@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 192.168.203.126@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg326-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing load_modules_local oleg326-server: Loading modules from /home/green/git/lustre-release/lustre oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg326-server: detected 4 online CPUs by sysfs oleg326-server: MODOPTS_LIBCFS= oleg326-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 192.168.203.126@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.126@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg326-client.virtnet /mnt/lustre (opts:) Stopping client oleg326-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing load_modules_local oleg326-server: Loading modules from /home/green/git/lustre-release/lustre oleg326-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg326-server: detected 4 online CPUs by sysfs oleg326-server: MODOPTS_LIBCFS= oleg326-server: Force libcfs to create 2 CPU partitions Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre2 PASS 109 (64s) == sanityn test 110: do not grant another lock on resend ========================================================== 01:10:15 (1787375415) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-22 01:10:17.000000000 -0400 Modify: 2026-08-22 01:10:17.000000000 -0400 Change: 2026-08-22 01:10:17.000000000 -0400 Birth: 2026-08-22 01:10:18.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg326-client.virtnet oleg326-client.virtnet PASS 110 (171s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 01:13:07 (1787375587) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 01:13:11 (1787375591) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 01:13:14 (1787375594) lustre-OST0000-osc-ffff9388faa7d800 active. lustre-OST0001-osc-ffff9388faa7d800 active. lustre-OST0000-osc-ffff9388faa7d800 active. lustre-OST0001-osc-ffff9388faa7d800 active. MGC192.168.203.126@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 01:13:21 (1787375601) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 01:13:24 (1787375604) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 01:13:27 (1787375607) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 01:13:30 (1787375610) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0120099 s, 1.7 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.091159 s, 225 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 200: service remains healthy while able to process request ========================================================== 01:13:39 (1787375619) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1348 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2143 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1696 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.138 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2086 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3068 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0421 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0568 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1647 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0879 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0896 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2182 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3689 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1694 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2384 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3262 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4237 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg326-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9388c3a7a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9388c3a7a800.ost_server_uuid in IDLE state after 0 sec PASS 200 (83s) cleanup: ====================================================== == sanityn test complete, duration 10171 sec ============= 01:15:03 (1787375703) === sanityn: start cleanup 01:15:05 (1787375705) === Stopping clients: oleg326-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg326-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 01:19:30 (1787375970) ===