-----============= acceptance-small: sanityn ============----- Sun Sep 6 16:25:12 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg450-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 16:25:38 (1788726338) === Starting client oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Mount client oleg450-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Started clients oleg450-client.virtnet: 192.168.204.150@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg450-client.virtnet: executing check_config_client /mnt/lustre oleg450-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg450-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90c144a07800.idle_timeout=debug osc.lustre-OST0000-osc-ffff90c15190c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90c144a07800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90c15190c800.idle_timeout=debug disable quota as required oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 16:26:20 (1788726380) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275244 s, 38.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7630] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 16:26:23 (1788726383) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 16:26:32 (1788726392) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 16:26:40 (1788726400) -rw-r--r-- 1 root root 0 Sep 6 16:26 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 16:26:48 (1788726408) -rw-r--r-- 1 root root 0 Sep 6 16:26 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 16:26:55 (1788726415) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 16:27:02 (1788726422) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (9s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 16:27:10 (1788726430) total 1 -rwxrwxrwx 1 root root 0 Sep 6 16:26 f2a -rwxrwxrwx 1 root root 0 Sep 6 16:26 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 6 16:26 f2a -rwxrwxrwx 1 root root 0 Sep 6 16:26 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0142413 s, 36.0 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 16:27:18 (1788726438) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 16:27:22 (1788726442) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.558104 s, 3.8 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 16:27:31 (1788726451) PASS 3 (9s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 16:27:39 (1788726459) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 16:27:47 (1788726467) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 16:27:55 (1788726475) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 16:28:04 (1788726484) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 16:28:11 (1788726491) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 16:28:18 (1788726498) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 16:28:26 (1788726506) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902764 s, 11.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0768045 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (9s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 16:28:35 (1788726515) 1+0 records in 1+0 records out 1 byte copied, 0.00846884 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.018339 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.014866 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0197621 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0408741 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0103588 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011024 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0149515 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0146342 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00877304 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0121606 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0189202 s, 0.1 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 16:28:44 (1788726524) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.059127 s, 52.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.083335 s, 49.2 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00250235 s, 1.2 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 16:28:53 (1788726533) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 16:29:00 (1788726540) warning: '-runtime' deprecated, use '-t runtime' instead - link 288 (time 1788726553.26 total 10.03 last 28.70) - link 559 (time 1788726563.30 total 20.07 last 27.00) - link 823 (time 1788726573.30 total 30.07 last 26.40) - link 1065 (time 1788726583.33 total 40.10 last 24.12) - link 1267 (time 1788726593.37 total 50.14 last 20.12) - link 1497 (time 1788726603.40 total 60.17 last 22.93) - link 1754 (time 1788726613.42 total 70.19 last 25.65) - link 2005 (time 1788726623.45 total 80.22 last 25.03) - link 2266 (time 1788726633.45 total 90.22 last 26.10) total: 2542 link in 100.03 seconds: 25.41 ops/second using seed 1950606184 running for 100 seconds - stat 4571 (time 1788726555 ; total 11 ; last 11) - stat 9011 (time 1788726566 ; total 22 ; last 11) - stat 10000 (time 1788726568 ; total 24 ; last 2) - stat 14656 (time 1788726579 ; total 35 ; last 11) - stat 19053 (time 1788726590 ; total 46 ; last 11) - stat 20000 (time 1788726593 ; total 49 ; last 3) - stat 22332 (time 1788726604 ; total 60 ; last 11) - stat 24661 (time 1788726615 ; total 71 ; last 11) - stat 26981 (time 1788726626 ; total 82 ; last 11) - stat 29223 (time 1788726637 ; total 93 ; last 11) - stat 30000 (time 1788726640 ; total 96 ; last 3) total: 30772 stats in 100 seconds: 307.720001 stats/second - unlinked 0 (time 1788726594 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2542) error: No such file or directory total: 2541 unlinks in 86 seconds: 29.546511 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21435) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21437) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 16:31:26 (1788726686) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 16:31:36 (1788726696) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22623 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 16:31:44 (1788726704) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23203 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 16:31:51 (1788726711) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23782 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 16:31:59 (1788726719) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24364 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 16:32:08 (1788726728) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24948 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 16:32:17 (1788726737) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (56s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 16:33:13 (1788726793) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.701619 s, 12.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.829302 s, 10.1 MB/s Chance of close/open is 1 in 50 Seed set to 6799 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x602bd0 000002[0] 1788726801.022113 trunc from 00000000 to 0x602bcf (0x602bd0 bytes) 000004[0] 1788726801.207761 punch from 0x584ea7 to 0x58a8f9 (0x5a53 bytes) 000006[1] 1788726801.267187 mapread 0x194b6c thru 0x195211 (0x6a6 bytes) 000007[1] 1788726801.335774 read 0x30d547 thru 0x31a8fd (0xd3b7 bytes) 000008[0] 1788726801.348434 read 0x5e5292 thru 0x5ecae8 (0x7857 bytes) 000009[0] 1788726801.358978 read 0x52abba thru 0x52e0e5 (0x352c bytes) 000011[1] 1788726801.383079 mapwrite 0x5248e5 thru 0x524d9b (0x4b7 bytes) 000012[1] 1788726801.527970 trunc from 0x602bd0 to 0x187b53 (0x47b07c bytes) 000014[0] 1788726801.674468 trunc from 0x187b54 to 0x356770 (0x1cec1d bytes) 000015[1] 1788726801.720336 write 0x72d017 thru 0x7364ab (0x9495 bytes) 000016[1] 1788726801.784330 trunc from 0x7364ac to 0x1c2488 (0x574023 bytes) 000017[0] 1788726801.892406 read 0x1c0ffd thru 0x1c2488 (0x148c bytes) 000019[0] 1788726802.010300 read 0x1bdde2 thru 0x1c2488 (0x46a7 bytes) 000021[0] 1788726802.048144 mapread 0x07b21f thru 0x084869 (0x964b bytes) truncating to largest ever: 0x7bcb7d 000022[0] 1788726802.085725 trunc from 0x1c2489 to 0x7bcb7c (0x5fa6f4 bytes) 000023[0] 1788726802.163191 mapwrite 0x7a1694 thru 0x7a6d9d (0x570a bytes) 000024[0] 1788726802.490299 punch from 0x41730c to 0x4212d5 (0x9fca bytes) 000025[0] 1788726802.519831 punch from 0x640588 to 0x643629 (0x30a2 bytes) 000030[1] 1788726802.605234 mapwrite 0x0e0bd4 thru 0x0e789c (0x6cc9 bytes) 000032[0] 1788726802.823899 write 0x2416da thru 0x2454ba (0x3de1 bytes) 000033[1] 1788726802.880162 mapread 0x528fb8 thru 0x52c27d (0x32c6 bytes) 000035[1] 1788726803.013722 write 0x2184cb thru 0x222ffd (0xab33 bytes) 000036[0] 1788726803.040406 punch from 0x6d5544 to 0x6d8051 (0x2b0e bytes) 000038[0] 1788726803.109254 mapread 0x325f9a thru 0x32cee8 (0x6f4f bytes) 000039[1] 1788726803.242245 mapread 0x4e5ecf thru 0x4e8af1 (0x2c23 bytes) 000040[1] 1788726803.286399 read 0x33ff7b thru 0x34962d (0x96b3 bytes) 000042[1] 1788726803.313015 mapread 0x2c7b04 thru 0x2d46ca (0xcbc7 bytes) 000046[0] 1788726803.394973 write 0x832839 thru 0x8352c0 (0x2a88 bytes) 000047[0] 1788726803.419546 trunc from 0x8352c1 to 0x6d47d0 (0x160af0 bytes) 000051[1] 1788726803.541268 read 0x22dda1 thru 0x2340d3 (0x6333 bytes) 000052[0] 1788726803.624852 mapwrite 0x54a224 thru 0x55650c (0xc2e9 bytes) 000053[0] 1788726803.890778 read 0x2f72a1 thru 0x304877 (0xd5d7 bytes) 000054[0] 1788726803.910908 mapread 0x05122f thru 0x05fcbd (0xea8f bytes) 000058[0] 1788726804.027999 punch from 0x3eea0c to 0x3fbe19 (0xd40e bytes) 000060[0] 1788726804.093859 mapread 0x3a613a thru 0x3a88f5 (0x27bc bytes) 000062[1] 1788726804.122009 mapread 0x51a73c thru 0x520f19 (0x67de bytes) truncating to largest ever: 0x8dfc46 000063[0] 1788726804.192696 trunc from 0x6d47d1 to 0x8dfc45 (0x20b475 bytes) 000065[0] 1788726804.248267 mapread 0x4f8331 thru 0x4fe9c7 (0x6697 bytes) 000066[0] 1788726804.271189 mapwrite 0x0a75e3 thru 0x0abd49 (0x4767 bytes) 000068[0] 1788726804.437937 read 0x05cf3c thru 0x06a14b (0xd210 bytes) 000069[0] 1788726804.490255 mapwrite 0x47950d thru 0x483bb8 (0xa6ac bytes) 000072[0] 1788726804.614240 punch from 0x4408df to 0x44167a (0xd9c bytes) 000073[1] 1788726804.662546 trunc from 0x8dfc46 to 0x026fc2 (0x8b8c83 bytes) 000076[1] 1788726804.810076 mapread 0x026bd6 thru 0x026fc2 (0x3ed bytes) 000077[0] 1788726804.822583 punch from 0x006fd3 to 0x00bd25 (0x4d53 bytes) 000078[0] 1788726804.868633 punch from 0x0052e9 to 0x011297 (0xbfaf bytes) 000080[0] 1788726804.913976 punch from 0x003ae5 to 0x00a8ac (0x6dc8 bytes) 000082[1] 1788726804.989320 mapread 0x007365 thru 0x00882a (0x14c6 bytes) 000086[1] 1788726805.123733 write 0x71b9c4 thru 0x7297a1 (0xddde bytes) truncating to largest ever: 0x9a97d9 000087[0] 1788726805.136095 trunc from 0x7297a2 to 0x9a97d8 (0x280037 bytes) 000089[0] 1788726805.266417 write 0x14aed8 thru 0x150569 (0x5692 bytes) 000090[1] 1788726805.285707 mapread 0x5ddf67 thru 0x5ecc8e (0xed28 bytes) 000092[1] 1788726805.470585 punch from 0x4da09f to 0x4de4f9 (0x445b bytes) 000093[0] 1788726805.510338 punch from 0x0104c7 to 0x01428c (0x3dc6 bytes) 000094[0] 1788726805.548719 mapwrite 0x138e74 thru 0x13912f (0x2bc bytes) 000096[0] 1788726805.756732 punch from 0x2ae9f7 to 0x2b2a80 (0x408a bytes) 000097[0] 1788726805.796033 punch from 0x437704 to 0x442c04 (0xb501 bytes) 000098[1] 1788726805.812789 read 0x1dce1b thru 0x1dd91e (0xb04 bytes) 000101[1] 1788726805.845305 mapread 0x6cdb3c thru 0x6d1e70 (0x4335 bytes) 000102[1] 1788726805.899645 write 0x942663 thru 0x94f6e6 (0xd084 bytes) 000104[0] 1788726805.940706 mapread 0x2a7e90 thru 0x2a80dc (0x24d bytes) 000105[1] 1788726806.052662 mapwrite 0x0d08d6 thru 0x0d26a1 (0x1dcc bytes) 000107[1] 1788726806.305689 punch from 0x7c59c3 to 0x7d0526 (0xab64 bytes) 000108[1] 1788726806.352914 mapwrite 0x71cde1 thru 0x71fc99 (0x2eb9 bytes) 000109[1] 1788726806.482191 mapread 0x420e86 thru 0x424ebc (0x4037 bytes) 000110[0] 1788726806.551039 mapread 0x23b582 thru 0x248822 (0xd2a1 bytes) 000113[0] 1788726806.709288 read 0x5c0605 thru 0x5cf650 (0xf04c bytes) 000114[1] 1788726806.742976 read 0x4c42ce thru 0x4d1da3 (0xdad6 bytes) 000115[1] 1788726806.766742 write 0x7280ea thru 0x728998 (0x8af bytes) 000116[1] 1788726806.851278 mapread 0x0543ec thru 0x060aa3 (0xc6b8 bytes) 000120[0] 1788726806.946145 punch from 0x1b1e97 to 0x1b846f (0x65d9 bytes) 000121[1] 1788726806.993758 punch from 0x8f2138 to 0x8f5abe (0x3987 bytes) 000122[0] 1788726807.092142 write 0x0b818b thru 0x0c38f3 (0xb769 bytes) 000123[0] 1788726807.141290 write 0x8ab4a3 thru 0x8b2efb (0x7a59 bytes) 000128[0] 1788726807.284331 write 0x919b06 thru 0x91fd8f (0x628a bytes) 000129[0] 1788726807.329792 read 0x2bfa31 thru 0x2c0620 (0xbf0 bytes) 000130[1] 1788726807.357093 mapread 0x7eab77 thru 0x7eea85 (0x3f0f bytes) 000131[1] 1788726807.458083 mapwrite 0x940375 thru 0x943d75 (0x3a01 bytes) 000132[1] 1788726807.619550 punch from 0x691ffe to 0x692959 (0x95c bytes) 000134[0] 1788726807.641081 read 0x68a103 thru 0x698e78 (0xed76 bytes) 000135[0] 1788726807.715456 trunc from 0x9a97d9 to 0x6f7f9a (0x2b183e bytes) 000137[0] 1788726807.798727 trunc from 0x6f7f9b to 0x7cf4e4 (0xd754a bytes) 000138[0] 1788726807.826209 mapwrite 0x32460d thru 0x32a0e6 (0x5ada bytes) 000140[1] 1788726808.146674 punch from 0x206112 to 0x21317a (0xd069 bytes) 000142[1] 1788726808.328696 trunc from 0x7cf4e5 to 0x366ea4 (0x468640 bytes) 000143[0] 1788726808.511318 trunc from 0x366ea5 to 0x813fa8 (0x4ad104 bytes) 000144[0] 1788726808.632904 mapwrite 0x0de8f5 thru 0x0ec856 (0xdf62 bytes) 000145[0] 1788726808.872885 read 0x2d817e thru 0x2de457 (0x62da bytes) 000147[0] 1788726808.943024 write 0x990205 thru 0x99d791 (0xd58d bytes) 000148[0] 1788726808.947087 punch from 0x326b0b to 0x334163 (0xd659 bytes) 000149[1] 1788726809.024086 write 0x554a9d thru 0x5581ea (0x374e bytes) 000152[1] 1788726809.146641 read 0x3a85fe thru 0x3b12f7 (0x8cfa bytes) 000153[0] 1788726809.245850 punch from 0x18c617 to 0x18d5d0 (0xfba bytes) 000158[0] 1788726809.473856 mapwrite 0x4bef6a thru 0x4bff68 (0xfff bytes) 000162[0] 1788726809.658697 trunc from 0x99d792 to 0x8db593 (0xc21fe bytes) 000164[0] 1788726809.693511 mapwrite 0x78b84f thru 0x78e857 (0x3009 bytes) 000165[0] 1788726809.850643 write 0x27e245 thru 0x283ee4 (0x5ca0 bytes) 000166[1] 1788726809.865502 mapread 0x277ad0 thru 0x27e516 (0x6a47 bytes) 000167[1] 1788726810.056186 read 0x783519 thru 0x78ccfd (0x97e5 bytes) 000168[0] 1788726810.100601 trunc from 0x8db594 to 0x39082a (0x54ad69 bytes) 000169[0] 1788726810.189961 read 0x25b23c thru 0x25d0d5 (0x1e9a bytes) 000170[1] 1788726810.340732 mapwrite 0x5f0ffc thru 0x5fcf46 (0xbf4b bytes) 000171[0] 1788726810.510650 mapread 0x3b6003 thru 0x3bb3a5 (0x53a3 bytes) 000176[0] 1788726810.843902 read 0x4fd29b thru 0x5043b9 (0x711f bytes) 000177[1] 1788726810.890268 mapread 0x356787 thru 0x35c4c0 (0x5d3a bytes) 000178[1] 1788726810.908800 write 0x071e2b thru 0x07702b (0x5201 bytes) 000179[1] 1788726810.952938 punch from 0x0a1491 to 0x0a826f (0x6ddf bytes) 000180[1] 1788726810.967178 punch from 0x00848f to 0x0169c4 (0xe536 bytes) 000183[0] 1788726810.980028 mapwrite 0x300dc2 thru 0x304eab (0x40ea bytes) 000184[0] 1788726811.195210 write 0x2d50dc thru 0x2ddbd0 (0x8af5 bytes) 000185[0] 1788726811.271908 mapwrite 0x73129e thru 0x73f506 (0xe269 bytes) 000186[0] 1788726811.517170 write 0x658ee6 thru 0x666e73 (0xdf8e bytes) 000187[1] 1788726811.536020 punch from 0x451260 to 0x460955 (0xf6f6 bytes) 000188[1] 1788726811.625278 read 0x0bd1e0 thru 0x0cbd0d (0xeb2e bytes) 000191[1] 1788726811.723291 read 0x2c25f4 thru 0x2c3e05 (0x1812 bytes) 000192[1] 1788726811.788980 mapread 0x4feada thru 0x5010f3 (0x261a bytes) 000193[1] 1788726811.862575 read 0x2080ac thru 0x20a885 (0x27da bytes) 000195[1] 1788726811.872806 mapread 0x5505a0 thru 0x55405c (0x3abd bytes) 000196[0] 1788726811.900491 trunc from 0x73f507 to 0x1ae407 (0x5910ff bytes) 000197[0] 1788726812.026360 read 0x1456db thru 0x14e227 (0x8b4d bytes) 000199[0] 1788726812.135221 mapwrite 0x1fa95e thru 0x20a09d (0xf740 bytes) 000200[1] 1788726812.271480 punch from 0x0c20a9 to 0x0d11dd (0xf135 bytes) 000203[0] 1788726812.361144 mapwrite 0x61df02 thru 0x62ded7 (0xffd6 bytes) 000204[1] 1788726812.670556 write 0x7df94d thru 0x7e566a (0x5d1e bytes) 000207[1] 1788726812.781172 trunc from 0x7e566b to 0x67db6b (0x167aff bytes) 000210[0] 1788726812.869214 punch from 0x0985f8 to 0x099125 (0xb2e bytes) 000212[0] 1788726812.918741 mapread 0x5232d9 thru 0x52bb90 (0x88b8 bytes) 000213[0] 1788726813.003086 punch from 0x3aff16 to 0x3b589c (0x5987 bytes) 000216[1] 1788726813.051438 read 0x65fd06 thru 0x66e97e (0xec79 bytes) 000217[1] 1788726813.084463 write 0x5479d3 thru 0x55756a (0xfb98 bytes) 000220[0] 1788726813.268838 mapread 0x406c5a thru 0x414df7 (0xe19e bytes) 000221[1] 1788726813.447184 punch from 0x5b6c24 to 0x5c4379 (0xd756 bytes) 000222[0] 1788726813.491759 write 0x9ce8f3 thru 0x9ced9d (0x4ab bytes) 000224[0] 1788726813.573587 mapread 0x527fc3 thru 0x53474f (0xc78d bytes) 000225[1] 1788726813.809142 mapwrite 0x6b28b0 thru 0x6c1559 (0xecaa bytes) 000226[1] 1788726814.525686 mapread 0x99b152 thru 0x9a94d8 (0xe387 bytes) 000227[0] 1788726814.631213 punch from 0x9725bf to 0x97b29a (0x8cdc bytes) 000229[0] 1788726814.690657 mapwrite 0x4527b1 thru 0x45773e (0x4f8e bytes) 000230[1] 1788726814.840757 punch from 0x1922f9 to 0x1924ba (0x1c2 bytes) 000232[1] 1788726814.919137 mapwrite 0x49614b thru 0x4a4702 (0xe5b8 bytes) 000234[1] 1788726815.120940 mapread 0x1e8b80 thru 0x1ee8ee (0x5d6f bytes) 000236[1] 1788726815.166913 write 0x0b175d thru 0x0b5445 (0x3ce9 bytes) 000240[0] 1788726815.336537 punch from 0x337fb1 to 0x343877 (0xb8c7 bytes) 000241[0] 1788726815.425672 write 0x99b693 thru 0x9a8b9f (0xd50d bytes) 000242[0] 1788726815.484257 trunc from 0x9ced9e to 0x0bcb4d (0x912250 bytes) 000243[0] 1788726815.526769 mapwrite 0x7be1b5 thru 0x7c23a6 (0x41f2 bytes) 000244[1] 1788726815.665193 write 0x599bc7 thru 0x599ea9 (0x2e3 bytes) 000246[0] 1788726815.743329 mapread 0x108a4c thru 0x1173ca (0xe97f bytes) 000247[0] 1788726815.864801 read 0x45c07e thru 0x46049f (0x4422 bytes) 000250[1] 1788726815.979882 write 0x44c474 thru 0x4587f0 (0xc37d bytes) 000251[0] 1788726816.086895 read 0x44eea4 thru 0x459c50 (0xadad bytes) 000254[1] 1788726816.205150 punch from 0x5cea61 to 0x5d621b (0x77bb bytes) 000255[0] 1788726816.267983 read 0x68cbe0 thru 0x692638 (0x5a59 bytes) 000257[1] 1788726816.377482 trunc from 0x7c23a7 to 0x01f666 (0x7a2d40 bytes) 000258[1] 1788726816.555823 punch from 0x0163f8 to 0x0197cb (0x33d4 bytes) 000261[0] 1788726816.605285 punch from 0x00a879 to 0x016b7f (0xc307 bytes) 000263[0] 1788726816.679241 trunc from 0x01f667 to 0x7c73e7 (0x7a7d81 bytes) 000264[0] 1788726816.749151 read 0x247102 thru 0x252eb3 (0xbdb2 bytes) 000270[0] 1788726817.003108 punch from 0x7c339d to 0x7c73e7 (0x404b bytes) 000271[0] 1788726817.058864 trunc from 0x7c73e8 to 0x834979 (0x6d592 bytes) 000273[0] 1788726817.104538 trunc from 0x83497a to 0x2c113f (0x57383a bytes) 000274[1] 1788726817.172286 mapwrite 0x63315f thru 0x640bc1 (0xda63 bytes) 000276[1] 1788726817.449079 write 0x169f4e thru 0x16e79d (0x4850 bytes) 000278[1] 1788726817.467218 write 0x36942d thru 0x377eba (0xea8e bytes) 000281[0] 1788726817.531775 read 0x21ee03 thru 0x21f141 (0x33f bytes) 000282[1] 1788726817.567486 mapwrite 0x51fb5e thru 0x529597 (0x9a3a bytes) 000284[1] 1788726817.706677 trunc from 0x640bc2 to 0x95cca2 (0x31c0e1 bytes) 000285[0] 1788726817.764047 mapwrite 0x56a6ad thru 0x57129f (0x6bf3 bytes) 000286[0] 1788726817.949255 read 0x4557ed thru 0x458251 (0x2a65 bytes) 000287[1] 1788726817.996246 mapwrite 0x37a866 thru 0x385dac (0xb547 bytes) 000289[1] 1788726818.240755 read 0x3876ba thru 0x395991 (0xe2d8 bytes) 000291[1] 1788726818.300873 trunc from 0x95cca3 to 0x6f47ed (0x2684b5 bytes) 000292[1] 1788726818.377608 write 0x24ee0f thru 0x258ec3 (0xa0b5 bytes) 000293[1] 1788726818.418711 mapwrite 0x993dc3 thru 0x9a2745 (0xe983 bytes) 000295[0] 1788726818.609012 trunc from 0x9a2746 to 0x204056 (0x79e6ef bytes) 000297[0] 1788726818.750159 write 0x5e6c75 thru 0x5f5a1a (0xeda6 bytes) 000299[0] 1788726818.848718 mapwrite 0x8b46bd thru 0x8c1718 (0xd05c bytes) 000300[1] 1788726819.038962 mapread 0x0c96f1 thru 0x0d3dac (0xa6bc bytes) 000301[0] 1788726819.114576 mapread 0x27db72 thru 0x28ac51 (0xd0e0 bytes) 000302[0] 1788726819.159010 mapwrite 0x3b2557 thru 0x3b6e29 (0x48d3 bytes) 000304[0] 1788726819.249855 mapread 0x7d591f thru 0x7e2a16 (0xd0f8 bytes) 000305[1] 1788726819.298954 write 0x90a652 thru 0x91299e (0x834d bytes) 000307[1] 1788726819.375011 trunc from 0x91299f to 0x45e9d9 (0x4b3fc5 bytes) 000308[0] 1788726819.439685 read 0x199440 thru 0x1a38e5 (0xa4a6 bytes) 000309[0] 1788726819.493770 mapwrite 0x03d893 thru 0x040d06 (0x3474 bytes) 000311[1] 1788726819.643670 mapwrite 0x054ae5 thru 0x06196b (0xce87 bytes) 000315[0] 1788726819.841824 trunc from 0x45e9da to 0x514eae (0xb64d5 bytes) 000316[0] 1788726819.895694 mapread 0x10cbaf thru 0x118979 (0xbdcb bytes) 000318[0] 1788726819.944763 trunc from 0x514eaf to 0x8b5565 (0x3a06b7 bytes) 000319[1] 1788726820.014773 mapread 0x3b777c thru 0x3c5171 (0xd9f6 bytes) 000320[1] 1788726820.143486 punch from 0x83f5e0 to 0x840c13 (0x1634 bytes) 000322[1] 1788726820.231550 punch from 0x1ec6fe to 0x1fabf6 (0xe4f9 bytes) 000324[0] 1788726820.280532 read 0x2e8a90 thru 0x2f81dc (0xf74d bytes) 000328[1] 1788726820.400356 write 0x4fe4ff thru 0x50990d (0xb40f bytes) 000329[1] 1788726820.425338 read 0x22a9bb thru 0x2315fb (0x6c41 bytes) 000330[1] 1788726820.466780 write 0x4e36cd thru 0x4f1b25 (0xe459 bytes) 000331[1] 1788726820.500001 read 0x135c9c thru 0x1414a1 (0xb806 bytes) 000334[0] 1788726820.596687 mapwrite 0x447855 thru 0x44beba (0x4666 bytes) 000335[1] 1788726820.874315 mapread 0x7e0b9f thru 0x7e5626 (0x4a88 bytes) 000336[0] 1788726820.976878 write 0x36d43f thru 0x37894e (0xb510 bytes) 000339[1] 1788726821.079811 punch from 0x0967a9 to 0x0a36d2 (0xcf2a bytes) 000340[0] 1788726821.152461 trunc from 0x8b5566 to 0x63e075 (0x2774f0 bytes) 000341[1] 1788726821.289037 mapread 0x26b3a5 thru 0x26c162 (0xdbe bytes) 000343[1] 1788726821.427752 mapwrite 0x32e537 thru 0x33d286 (0xed50 bytes) 000347[0] 1788726821.733996 mapread 0x0e2018 thru 0x0e38de (0x18c7 bytes) 000348[1] 1788726821.785751 read 0x302fad thru 0x3044b0 (0x1504 bytes) 000350[0] 1788726821.809724 read 0x53f90f thru 0x545ea5 (0x6597 bytes) 000352[1] 1788726821.855169 read 0x1938cd thru 0x195d95 (0x24c9 bytes) 000355[0] 1788726821.915627 mapread 0x04b918 thru 0x04f2d4 (0x39bd bytes) 000356[0] 1788726821.950671 write 0x2cf42a thru 0x2dabe9 (0xb7c0 bytes) 000357[0] 1788726821.983503 write 0x771b68 thru 0x773008 (0x14a1 bytes) 000358[1] 1788726821.988316 mapread 0x710a20 thru 0x71bba3 (0xb184 bytes) 000359[1] 1788726822.056697 punch from 0x11774e to 0x11ce60 (0x5713 bytes) 000360[1] 1788726822.105488 read 0x6504de thru 0x6505c3 (0xe6 bytes) 000361[1] 1788726822.127185 punch from 0x1f35c4 to 0x1f796d (0x43aa bytes) 000364[0] 1788726822.149604 read 0x0925ca thru 0x09fea0 (0xd8d7 bytes) 000365[0] 1788726822.182065 mapwrite 0x84a9b6 thru 0x853b6a (0x91b5 bytes) 000366[1] 1788726822.419967 trunc from 0x853b6b to 0x4e593d (0x36e22d bytes) 000367[0] 1788726822.504993 punch from 0x1d7652 to 0x1e65d5 (0xef84 bytes) 000368[0] 1788726822.553172 read 0x08cac0 thru 0x08fd88 (0x32c9 bytes) 000369[1] 1788726822.587734 trunc from 0x4e593e to 0x24c1e1 (0x29975c bytes) 000373[1] 1788726822.743112 mapwrite 0x067a69 thru 0x070b46 (0x90de bytes) 000374[0] 1788726822.889776 write 0x3c42e8 thru 0x3caa5c (0x6775 bytes) 000375[1] 1788726822.937058 punch from 0x0573b1 to 0x05ee74 (0x7ac4 bytes) 000376[0] 1788726822.973904 trunc from 0x3caa5d to 0x87eba9 (0x4b414d bytes) 000377[1] 1788726823.137361 trunc from 0x87ebaa to 0x1c4c83 (0x6b9f26 bytes) 000378[1] 1788726823.287003 mapread 0x1b4c7f thru 0x1bc8d7 (0x7c59 bytes) 000379[0] 1788726823.354066 mapwrite 0x729708 thru 0x72a4b3 (0xdac bytes) 000380[0] 1788726823.593898 trunc from 0x72a4b4 to 0x18b11d (0x59f396 bytes) 000381[0] 1788726823.641934 trunc from 0x18b11e to 0x556aab (0x3cb98e bytes) 000382[1] 1788726823.696605 punch from 0x30d1c6 to 0x312af2 (0x592d bytes) 000383[0] 1788726823.744868 read 0x180970 thru 0x18f81f (0xeeb0 bytes) 000386[0] 1788726823.845794 mapread 0x289558 thru 0x289b7a (0x623 bytes) 000388[0] 1788726823.851796 write 0x06103c thru 0x07084d (0xf812 bytes) 000390[0] 1788726823.931863 mapwrite 0x8046e6 thru 0x807f14 (0x382f bytes) 000391[0] 1788726824.154857 trunc from 0x807f15 to 0x81c000 (0x140ec bytes) 000392[1] 1788726824.266101 write 0x11eda5 thru 0x12ebca (0xfe26 bytes) 000393[0] 1788726824.372908 punch from 0x7f301c to 0x7f5306 (0x22eb bytes) 000394[1] 1788726824.421645 mapwrite 0x866781 thru 0x86bad0 (0x5350 bytes) 000396[1] 1788726824.570701 mapread 0x5f8812 thru 0x6010b3 (0x88a2 bytes) 000398[0] 1788726824.639263 write 0x12963a thru 0x136097 (0xca5e bytes) 000400[1] 1788726824.820774 punch from 0x0cd3ee to 0x0d371e (0x6331 bytes) 000401[1] 1788726824.889160 write 0x0aa3ec thru 0x0b0b24 (0x6739 bytes) 000402[0] 1788726824.945730 write 0x461e44 thru 0x470d01 (0xeebe bytes) 000404[0] 1788726825.027674 read 0x1fcafd thru 0x20518e (0x8692 bytes) 000405[0] 1788726825.086650 mapwrite 0x7fc5aa thru 0x8059bb (0x9412 bytes) 000408[0] 1788726825.352641 trunc from 0x86bad1 to 0x05b414 (0x8106bc bytes) 000409[0] 1788726825.441514 mapread 0x0499df thru 0x05500d (0xb62f bytes) 000410[0] 1788726825.547816 read 0x0116ab thru 0x015361 (0x3cb7 bytes) 000413[1] 1788726825.694195 write 0x678a0b thru 0x67cf5c (0x4552 bytes) 000414[0] 1788726825.755010 mapread 0x2e20cc thru 0x2e6d7c (0x4cb1 bytes) 000416[0] 1788726825.953745 write 0x8e8e1a thru 0x8f8772 (0xf959 bytes) 000418[0] 1788726826.006223 read 0x5e4ec2 thru 0x5ef69c (0xa7db bytes) 000420[1] 1788726826.053221 read 0x024340 thru 0x03052e (0xc1ef bytes) 000421[0] 1788726826.099921 read 0x6ee17f thru 0x6f4e7e (0x6d00 bytes) 000422[1] 1788726826.135024 punch from 0x83bcb2 to 0x841c1e (0x5f6d bytes) 000423[1] 1788726826.150833 trunc from 0x8f8773 to 0x248ec8 (0x6af8aa bytes) 000424[1] 1788726826.231551 punch from 0x0ede94 to 0x0fb721 (0xd88e bytes) 000425[0] 1788726826.297313 read 0x03eb25 thru 0x04abae (0xc08a bytes) 000426[1] 1788726826.418581 read 0x0f4c96 thru 0x1009f4 (0xbd5f bytes) 000427[1] 1788726826.546186 trunc from 0x248ec9 to 0x27f126 (0x3625e bytes) 000429[1] 1788726826.700927 punch from 0x1dc335 to 0x1e0e22 (0x4aee bytes) 000430[1] 1788726826.765932 write 0x02ac00 thru 0x037c55 (0xd056 bytes) 000433[0] 1788726826.836651 mapwrite 0x9ef94a thru 0x9f8e4f (0x9506 bytes) 000434[0] 1788726827.192517 trunc from 0x9f8e50 to 0x05c03d (0x99ce12 bytes) 000435[0] 1788726827.232144 write 0x45ffc9 thru 0x46669c (0x66d4 bytes) 000436[1] 1788726827.258183 trunc from 0x46669d to 0x2500a6 (0x2165f6 bytes) 000439[0] 1788726827.403117 write 0x32ba0c thru 0x3357e3 (0x9dd8 bytes) 000440[0] 1788726827.446942 trunc from 0x3357e4 to 0x11db2e (0x217cb5 bytes) 000442[1] 1788726827.531994 read 0x0b83ac thru 0x0c050e (0x8163 bytes) 000443[1] 1788726827.655993 read 0x031f82 thru 0x035270 (0x32ef bytes) 000444[0] 1788726827.685419 mapwrite 0x33907a thru 0x34773a (0xe6c1 bytes) 000445[0] 1788726828.122352 punch from 0x341ed9 to 0x3449a0 (0x2ac8 bytes) 000446[1] 1788726828.173048 mapread 0x287dd4 thru 0x294109 (0xc336 bytes) 000447[0] 1788726828.271045 read 0x30b6d9 thru 0x30d53e (0x1e66 bytes) 000448[0] 1788726828.339773 mapread 0x23dd5e thru 0x2469b7 (0x8c5a bytes) 000449[1] 1788726828.341488 read 0x137b96 thru 0x147446 (0xf8b1 bytes) 000450[0] 1788726828.342302 mapread 0x0acc09 thru 0x0ae0f7 (0x14ef bytes) 000451[1] 1788726828.366884 read 0x0c0af2 thru 0x0c7212 (0x6721 bytes) 000454[1] 1788726828.404061 read 0x27d59d thru 0x28b076 (0xdada bytes) 000455[0] 1788726828.437548 mapread 0x309f60 thru 0x30cf00 (0x2fa1 bytes) 000456[0] 1788726828.438462 punch from 0x06c039 to 0x073442 (0x740a bytes) 000457[1] 1788726828.508689 write 0x2a3bcf thru 0x2a7493 (0x38c5 bytes) 000459[1] 1788726828.657689 trunc from 0x34773b to 0x6a0bd5 (0x35949b bytes) 000460[0] 1788726828.704802 read 0x066821 thru 0x071476 (0xac56 bytes) 000461[0] 1788726828.840654 read 0x42734b thru 0x428d78 (0x1a2e bytes) 000463[1] 1788726828.862339 mapwrite 0x028ead thru 0x0336c8 (0xa81c bytes) 000464[1] 1788726829.172134 write 0x17f91d thru 0x18143d (0x1b21 bytes) 000465[0] 1788726829.191138 write 0x877bfd thru 0x87905d (0x1461 bytes) 000466[0] 1788726829.245436 read 0x144b5e thru 0x14c722 (0x7bc5 bytes) 000467[1] 1788726829.265964 trunc from 0x87905e to 0x08b3fe (0x7edc5f bytes) 000468[1] 1788726829.377996 mapread 0x02fbcf thru 0x03e6f6 (0xeb28 bytes) 000471[0] 1788726829.501448 write 0x160d40 thru 0x170795 (0xfa56 bytes) 000472[0] 1788726829.556605 trunc from 0x170796 to 0x708119 (0x597984 bytes) 000474[1] 1788726829.597947 read 0x5daea3 thru 0x5e7e27 (0xcf85 bytes) 000476[0] 1788726829.809125 read 0x1e0149 thru 0x1e2e51 (0x2d09 bytes) 000477[1] 1788726829.818754 write 0x8e8298 thru 0x8f7362 (0xf0cb bytes) 000478[1] 1788726829.882133 write 0x86f86e thru 0x872335 (0x2ac8 bytes) 000480[0] 1788726830.079541 read 0x6e8da5 thru 0x6f1ce4 (0x8f40 bytes) 000482[1] 1788726830.096851 read 0x40b7a3 thru 0x4132cc (0x7b2a bytes) 000484[1] 1788726830.137197 read 0x09b825 thru 0x0a92e4 (0xdac0 bytes) 000486[1] 1788726830.203043 trunc from 0x8f7363 to 0x43e0c9 (0x4b9299 bytes) 000487[1] 1788726830.252059 mapwrite 0x19d0b9 thru 0x1a109a (0x3fe2 bytes) 000488[0] 1788726830.368096 mapread 0x267b48 thru 0x275d14 (0xe1cd bytes) 000489[0] 1788726830.500017 mapread 0x1d0320 thru 0x1d77e9 (0x74ca bytes) 000490[1] 1788726830.530856 write 0x269da4 thru 0x27556b (0xb7c8 bytes) 000491[0] 1788726830.613261 punch from 0x20e88b to 0x21a61c (0xbd92 bytes) 000493[1] 1788726830.727675 read 0x343876 thru 0x34b478 (0x7c03 bytes) 000496[0] 1788726830.868597 mapread 0x36f923 thru 0x378259 (0x8937 bytes) 000499[1] 1788726830.975418 read 0x216f1e thru 0x2187b4 (0x1897 bytes) 000500[1] 1788726830.984338 read 0x14e709 thru 0x155ead (0x77a5 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 6831 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1788726831.794811 mapwrite 0x636000 thru 0x636b84 (0xb85 bytes) 000004[0] 1788726831.944115 punch from 0x3c4177 to 0x3d21f8 (0xe082 bytes) 000005[1] 1788726831.970206 read 0x20a000 thru 0x216e5b (0xce5c bytes) truncating to largest ever: 0x1aad4f 000007[0] 1788726831.998852 trunc from 0x636b85 to 0x1aad4e (0x48be36 bytes) 000010[0] 1788726832.064940 punch from 0x170bc2 to 0x180a26 (0xfe65 bytes) 000012[1] 1788726832.071745 mapread 0x042000 thru 0x050224 (0xe225 bytes) 000014[0] 1788726832.167997 mapwrite 0x823000 thru 0x82e589 (0xb58a bytes) 000019[0] 1788726832.461087 punch from 0x3a6632 to 0x3a9015 (0x29e4 bytes) 000023[1] 1788726832.483064 read 0x067000 thru 0x07404e (0xd04f bytes) 000024[0] 1788726832.504930 mapread 0x05a000 thru 0x05a0bb (0xbc bytes) 000025[0] 1788726832.513335 punch from 0x66c6e4 to 0x66c718 (0x35 bytes) 000026[0] 1788726832.562220 mapwrite 0x125000 thru 0x130d3e (0xbd3f bytes) 000027[0] 1788726832.710205 mapwrite 0x711000 thru 0x71b66a (0xa66b bytes) 000028[0] 1788726832.933233 mapwrite 0x0ac000 thru 0x0bb03a (0xf03b bytes) 000030[0] 1788726833.124609 punch from 0x5e8283 to 0x5f2732 (0xa4b0 bytes) 000031[1] 1788726833.170146 mapwrite 0x0db000 thru 0x0df41a (0x441b bytes) 000032[1] 1788726833.429453 read 0x32e000 thru 0x33941a (0xb41b bytes) 000034[1] 1788726833.496328 mapwrite 0x82c000 thru 0x82c5e2 (0x5e3 bytes) 000036[1] 1788726833.607486 mapwrite 0x2ed000 thru 0x2f21c0 (0x51c1 bytes) 000040[0] 1788726833.748781 mapwrite 0x9b3000 thru 0x9bb003 (0x8004 bytes) 000042[0] 1788726833.897908 read_OD 0x38b000 thru 0x397fff (0xd000 bytes) 000043[1] 1788726833.927503 write 0x0a8000 thru 0x0aed20 (0x6d21 bytes) truncating to largest ever: 0x2da68b 000046[1] 1788726834.002083 trunc from 0x9bb004 to 0x2da68a (0x6e0979 bytes) 000047[0] 1788726834.044642 write_OD 0x50a000 thru 0x50cfff (0x3000 bytes) 000049[0] 1788726834.235331 punch from 0x143940 to 0x145a3f (0x2100 bytes) 000052[1] 1788726834.283411 mapread 0x44f000 thru 0x454cfe (0x5cff bytes) 000053[1] 1788726834.378261 write 0x6be000 thru 0x6be5c8 (0x5c9 bytes) 000055[1] 1788726834.458018 mapread 0x3dc000 thru 0x3eb2c4 (0xf2c5 bytes) 000056[0] 1788726834.510101 write_OD 0x86d000 thru 0x86ffff (0x3000 bytes) 000057[1] 1788726834.607142 mapread 0x633000 thru 0x63744e (0x444f bytes) 000058[0] 1788726834.680783 write_OD 0x080000 thru 0x080fff (0x1000 bytes) 000059[0] 1788726834.794624 write_OD 0x1e5000 thru 0x1e5fff (0x1000 bytes) 000060[1] 1788726834.854692 punch from 0x2068c1 to 0x20a701 (0x3e41 bytes) 000061[0] 1788726834.884341 mapwrite 0x2dd000 thru 0x2eb761 (0xe762 bytes) 000062[0] 1788726835.120689 punch from 0x305ea2 to 0x30dc44 (0x7da3 bytes) 000064[1] 1788726835.188918 mapwrite 0x30b000 thru 0x31065c (0x565d bytes) 000065[0] 1788726835.412686 mapread 0x5b0000 thru 0x5b1ac4 (0x1ac5 bytes) 000066[1] 1788726835.466311 punch from 0x7e2b00 to 0x7efdf4 (0xd2f5 bytes) 000067[0] 1788726835.496643 read_OD 0x4bf000 thru 0x4c9fff (0xb000 bytes) 000070[1] 1788726835.557177 mapread 0x34b000 thru 0x357977 (0xc978 bytes) 000073[0] 1788726835.625410 punch from 0x780232 to 0x7866d1 (0x64a0 bytes) truncating to largest ever: 0x6658ed 000075[1] 1788726835.753815 trunc from 0x870000 to 0x6658ec (0x20a713 bytes) 000076[1] 1788726835.877444 mapwrite 0x3b8000 thru 0x3be5bd (0x65be bytes) 000077[1] 1788726836.078315 write 0x754000 thru 0x75a4b8 (0x64b9 bytes) 000078[0] 1788726836.115089 write_OD 0x6a5000 thru 0x6b0fff (0xc000 bytes) 000079[1] 1788726836.280945 mapwrite 0x891000 thru 0x893ece (0x2ecf bytes) 000080[1] 1788726836.449315 trunc from 0x893ecf to 0x2ddaf8 (0x5b63d6 bytes) 000081[0] 1788726836.475822 mapread 0x24b000 thru 0x24e6b0 (0x36b1 bytes) 000082[1] 1788726836.535973 write 0x6ec000 thru 0x6efcd4 (0x3cd5 bytes) 000083[0] 1788726836.561656 mapread 0x59b000 thru 0x5a9202 (0xe203 bytes) 000085[1] 1788726836.697317 mapread 0x02c000 thru 0x033b9a (0x7b9b bytes) 000087[0] 1788726836.742886 punch from 0x1c7eae to 0x1c8a0f (0xb62 bytes) 000092[1] 1788726836.832353 punch from 0x0d7722 to 0x0d8c07 (0x14e6 bytes) 000093[1] 1788726836.872946 punch from 0x51d6c0 to 0x521744 (0x4085 bytes) truncating to largest ever: 0x8a5246 000094[0] 1788726836.910339 trunc from 0x6efcd5 to 0x8a5245 (0x1b5571 bytes) 000097[1] 1788726836.954003 read 0x072000 thru 0x0813c3 (0xf3c4 bytes) 000098[1] 1788726836.988343 punch from 0x5aa28a to 0x5b0d67 (0x6ade bytes) 000099[1] 1788726837.019734 mapread 0x65f000 thru 0x66a068 (0xb069 bytes) 000101[1] 1788726837.137088 mapread 0x5b1000 thru 0x5b1714 (0x715 bytes) 000103[1] 1788726837.169484 mapwrite 0x7e8000 thru 0x7ea161 (0x2162 bytes) 000104[0] 1788726837.291758 trunc from 0x8a5246 to 0x244a1a (0x66082b bytes) 000107[1] 1788726837.400258 write 0x70f000 thru 0x71a145 (0xb146 bytes) 000108[1] 1788726837.444783 read 0x260000 thru 0x26471a (0x471b bytes) 000109[0] 1788726837.485777 write_OD 0x90c000 thru 0x913fff (0x8000 bytes) 000111[0] 1788726837.617176 mapread 0x03b000 thru 0x045d03 (0xad04 bytes) 000113[1] 1788726837.664602 trunc from 0x914000 to 0x5141af (0x3ffe50 bytes) 000114[0] 1788726837.791473 trunc from 0x5141b0 to 0x0bbb1e (0x458691 bytes) 000115[0] 1788726837.842978 mapwrite 0x88d000 thru 0x88e356 (0x1357 bytes) 000117[1] 1788726838.089209 trunc from 0x88e357 to 0x1be554 (0x6cfe02 bytes) 000118[1] 1788726838.126148 read 0x01f000 thru 0x028238 (0x9239 bytes) 000119[1] 1788726838.226294 write 0x2d4000 thru 0x2e26be (0xe6bf bytes) 000121[0] 1788726838.274634 trunc from 0x2e26bf to 0x54d89a (0x26b1dc bytes) 000122[0] 1788726838.382160 punch from 0x50b6a1 to 0x517d20 (0xc680 bytes) 000123[1] 1788726838.441959 read 0x038000 thru 0x047494 (0xf495 bytes) 000126[0] 1788726838.473402 mapread 0x239000 thru 0x23e8fd (0x58fe bytes) 000127[0] 1788726838.494176 write_OD 0x5df000 thru 0x5e2fff (0x4000 bytes) 000128[1] 1788726838.563045 trunc from 0x5e3000 to 0x77ed98 (0x19bd99 bytes) 000129[1] 1788726838.640852 mapread 0x279000 thru 0x27a44d (0x144e bytes) 000130[1] 1788726838.663255 mapwrite 0x394000 thru 0x398e9d (0x4e9e bytes) 000131[1] 1788726838.755338 trunc from 0x77ed99 to 0x1200cb (0x65eccd bytes) 000132[1] 1788726838.782970 mapwrite 0x772000 thru 0x7733d0 (0x13d1 bytes) 000134[0] 1788726838.983044 write_OD 0x0e6000 thru 0x0f3fff (0xe000 bytes) 000135[0] 1788726839.108644 mapread 0x1ca000 thru 0x1d22b1 (0x82b2 bytes) 000136[0] 1788726839.125632 write_OD 0x1c9000 thru 0x1d1fff (0x9000 bytes) 000139[0] 1788726839.190588 write_OD 0x3e1000 thru 0x3e2fff (0x2000 bytes) 000140[1] 1788726839.280605 read 0x021000 thru 0x02fbed (0xebee bytes) 000142[1] 1788726839.314607 mapread 0x02e000 thru 0x03b54c (0xd54d bytes) 000145[0] 1788726839.387661 mapread 0x53b000 thru 0x546881 (0xb882 bytes) 000146[0] 1788726839.443072 mapwrite 0x4e2000 thru 0x4e4ac2 (0x2ac3 bytes) 000147[1] 1788726839.582227 mapread 0x24a000 thru 0x25994c (0xf94d bytes) 000149[1] 1788726839.635395 write 0x3a5000 thru 0x3b134d (0xc34e bytes) 000151[1] 1788726839.677721 mapread 0x10a000 thru 0x10cc4e (0x2c4f bytes) 000153[1] 1788726839.705446 trunc from 0x7733d1 to 0x07ae90 (0x6f8540 bytes) 000154[1] 1788726839.731264 mapread 0x062000 thru 0x06b5d3 (0x95d4 bytes) 000155[1] 1788726839.751293 mapread 0x073000 thru 0x076db2 (0x3db3 bytes) 000157[1] 1788726839.785613 write 0x170000 thru 0x17b925 (0xb926 bytes) 000158[0] 1788726839.788181 read_OD 0x09e000 thru 0x0abfff (0xe000 bytes) 000160[1] 1788726839.832050 mapwrite 0x685000 thru 0x693240 (0xe241 bytes) 000161[1] 1788726840.058307 punch from 0x5500ad to 0x55a3cc (0xa320 bytes) 000163[1] 1788726840.099194 trunc from 0x693241 to 0x37c517 (0x316d29 bytes) 000164[1] 1788726840.174968 punch from 0x21f181 to 0x221e55 (0x2cd5 bytes) 000165[1] 1788726840.218621 read 0x164000 thru 0x168f7a (0x4f7b bytes) 000167[0] 1788726840.368054 write_OD 0x18a000 thru 0x18cfff (0x3000 bytes) 000170[0] 1788726840.511740 trunc from 0x37c518 to 0x3e8e3d (0x6c926 bytes) 000171[0] 1788726840.585319 mapwrite 0x21b000 thru 0x21b9fd (0x9fe bytes) 000172[0] 1788726840.781065 mapread 0x1eb000 thru 0x1f9259 (0xe25a bytes) 000176[1] 1788726840.872500 read 0x3db000 thru 0x3e070b (0x570c bytes) 000180[0] 1788726841.001193 mapread 0x16f000 thru 0x17e7ad (0xf7ae bytes) 000181[0] 1788726841.080705 mapread 0x2ff000 thru 0x306e59 (0x7e5a bytes) 000183[0] 1788726841.117046 read_OD 0x05a000 thru 0x064fff (0xb000 bytes) 000185[1] 1788726841.126000 punch from 0x16fd63 to 0x170d2f (0xfcd bytes) 000187[1] 1788726841.181789 punch from 0x33790a to 0x3437d2 (0xbec9 bytes) 000189[0] 1788726841.218585 mapwrite 0x9d6000 thru 0x9dbb9f (0x5ba0 bytes) 000191[1] 1788726841.688313 mapwrite 0x02e000 thru 0x032538 (0x4539 bytes) 000193[1] 1788726841.835488 read 0x410000 thru 0x41841b (0x841c bytes) 000194[0] 1788726841.846972 mapwrite 0x6a4000 thru 0x6accf1 (0x8cf2 bytes) 000196[1] 1788726842.057636 read 0x017000 thru 0x01dc02 (0x6c03 bytes) 000197[0] 1788726842.092930 trunc from 0x9dbba0 to 0x208e6b (0x7d2d34 bytes) 000198[0] 1788726842.139547 mapwrite 0x1d8000 thru 0x1d96f7 (0x16f8 bytes) 000199[1] 1788726842.203674 trunc from 0x208e6c to 0x1d4dfd (0x3406e bytes) truncating to largest ever: 0x94f4df 000200[1] 1788726842.242190 trunc from 0x1d4dfe to 0x94f4de (0x77a6e1 bytes) 000202[0] 1788726842.297195 punch from 0x8f0ebe to 0x8f9ea9 (0x8fec bytes) 000208[0] 1788726842.406619 read_OD 0x199000 thru 0x1a4fff (0xc000 bytes) 000209[1] 1788726842.435140 read 0x463000 thru 0x4722cd (0xf2ce bytes) 000211[0] 1788726842.458799 punch from 0x4188ed to 0x41b9db (0x30ef bytes) 000212[0] 1788726842.521181 write_OD 0x085000 thru 0x085fff (0x1000 bytes) 000214[1] 1788726842.599692 trunc from 0x94f4df to 0x3b3a25 (0x59bab9 bytes) 000215[0] 1788726842.635476 mapwrite 0x8d8000 thru 0x8e74e3 (0xf4e4 bytes) 000216[1] 1788726842.804614 write 0x2ab000 thru 0x2b6f4d (0xbf4e bytes) 000217[0] 1788726842.842983 punch from 0x855472 to 0x860ada (0xb669 bytes) 000219[1] 1788726842.895499 mapread 0x56f000 thru 0x576b89 (0x7b8a bytes) 000220[0] 1788726842.939916 read_OD 0x7fb000 thru 0x7fefff (0x4000 bytes) 000221[1] 1788726842.948188 read 0x51d000 thru 0x52b866 (0xe867 bytes) 000222[0] 1788726842.957309 punch from 0x7f7eb0 to 0x8006af (0x8800 bytes) 000223[1] 1788726842.976697 write 0x61d000 thru 0x621cbb (0x4cbc bytes) 000225[0] 1788726843.014846 mapwrite 0x77c000 thru 0x78489c (0x889d bytes) 000228[0] 1788726843.234595 mapread 0x717000 thru 0x71ff34 (0x8f35 bytes) 000230[0] 1788726843.270116 write_OD 0x170000 thru 0x17efff (0xf000 bytes) 000233[1] 1788726843.342434 mapread 0x57d000 thru 0x583d4d (0x6d4e bytes) 000234[1] 1788726843.358537 mapread 0x7af000 thru 0x7b7c92 (0x8c93 bytes) 000235[1] 1788726843.384809 mapread 0x2f4000 thru 0x303457 (0xf458 bytes) 000237[0] 1788726843.421100 mapwrite 0x9dc000 thru 0x9e9236 (0xd237 bytes) 000238[1] 1788726843.613793 punch from 0x1f0741 to 0x1f8927 (0x81e7 bytes) 000239[0] 1788726843.666234 write_OD 0x75b000 thru 0x767fff (0xd000 bytes) 000240[0] 1788726843.783146 trunc from 0x9e9237 to 0x510e6c (0x4d83ca bytes) 000242[0] 1788726843.869147 mapwrite 0x6a4000 thru 0x6ae5ee (0xa5ef bytes) 000243[1] 1788726844.037687 write 0x141000 thru 0x14fd32 (0xed33 bytes) 000245[1] 1788726844.090680 trunc from 0x6ae5ef to 0x520e69 (0x18d785 bytes) 000247[1] 1788726844.187935 mapwrite 0x120000 thru 0x12a3ae (0xa3af bytes) 000249[0] 1788726844.331221 read_OD 0x3be000 thru 0x3c9fff (0xc000 bytes) 000250[1] 1788726844.364639 mapread 0x182000 thru 0x184bdd (0x2bde bytes) 000251[1] 1788726844.379971 trunc from 0x520e6a to 0x79d51b (0x27c6b2 bytes) 000253[0] 1788726844.481091 read_OD 0x5b1000 thru 0x5b9fff (0x9000 bytes) 000254[1] 1788726844.529987 mapread 0x405000 thru 0x4065aa (0x15ab bytes) 000257[0] 1788726844.551340 trunc from 0x79d51c to 0x5b57d1 (0x1e7d4a bytes) 000258[0] 1788726844.582224 mapread 0x136000 thru 0x1393d1 (0x33d2 bytes) 000260[1] 1788726844.663539 read 0x109000 thru 0x10c8b7 (0x38b8 bytes) 000261[1] 1788726844.691964 punch from 0x344dc7 to 0x34500d (0x247 bytes) 000262[0] 1788726844.730452 punch from 0x2798b3 to 0x287f69 (0xe6b7 bytes) 000264[1] 1788726844.752061 read 0x21e000 thru 0x229da6 (0xbda7 bytes) 000265[1] 1788726844.780229 trunc from 0x5b57d2 to 0x27590c (0x33fec5 bytes) 000266[1] 1788726844.839869 mapread 0x08f000 thru 0x09314b (0x414c bytes) 000269[1] 1788726844.908856 trunc from 0x27590d to 0x3dcba6 (0x16729a bytes) 000270[1] 1788726844.934623 trunc from 0x3dcba7 to 0x3f738e (0x1a7e8 bytes) 000273[1] 1788726844.997124 mapwrite 0x3d8000 thru 0x3e63d2 (0xe3d3 bytes) 000275[0] 1788726845.113943 punch from 0x0db57f to 0x0e7074 (0xbaf6 bytes) 000276[1] 1788726845.133232 mapread 0x284000 thru 0x28b4e2 (0x74e3 bytes) 000277[1] 1788726845.173184 trunc from 0x3f738f to 0x199c2f (0x25d75f bytes) 000279[1] 1788726845.226578 punch from 0x0b77b4 to 0x0c28b3 (0xb100 bytes) truncating to largest ever: 0x9c8dbc 000281[1] 1788726845.292026 trunc from 0x199c30 to 0x9c8dbb (0x82f18c bytes) 000282[1] 1788726845.337806 write 0x743000 thru 0x74e8ff (0xb900 bytes) 000283[1] 1788726845.379552 read 0x6e2000 thru 0x6eb20f (0x9210 bytes) 000284[1] 1788726845.426146 read 0x908000 thru 0x916879 (0xe87a bytes) 000286[0] 1788726845.445174 mapwrite 0x895000 thru 0x89710f (0x2110 bytes) 000288[0] 1788726845.594939 mapread 0x587000 thru 0x594519 (0xd51a bytes) 000291[1] 1788726845.668004 mapwrite 0x7f2000 thru 0x7fa632 (0x8633 bytes) 000294[0] 1788726845.949626 mapwrite 0x31b000 thru 0x32628a (0xb28b bytes) 000297[1] 1788726846.210774 punch from 0x32cbfa to 0x333a6e (0x6e75 bytes) 000299[1] 1788726846.250265 punch from 0x9000e0 to 0x902c4d (0x2b6e bytes) 000301[0] 1788726846.289669 read_OD 0x31f000 thru 0x32bfff (0xd000 bytes) 000303[0] 1788726846.321952 trunc from 0x9c8dbc to 0x5a0e14 (0x427fa7 bytes) 000304[1] 1788726846.369349 read 0x17d000 thru 0x181ae8 (0x4ae9 bytes) 000306[1] 1788726846.406857 mapread 0x3a8000 thru 0x3a9eeb (0x1eec bytes) 000308[1] 1788726846.429812 read 0x29e000 thru 0x2a9912 (0xb913 bytes) 000309[1] 1788726846.459618 punch from 0x11f9a8 to 0x11fa89 (0xe2 bytes) 000310[1] 1788726846.516113 mapwrite 0x954000 thru 0x95e3c7 (0xa3c8 bytes) 000314[1] 1788726846.743127 mapread 0x34e000 thru 0x352285 (0x4286 bytes) 000315[1] 1788726846.784139 punch from 0x097c0b to 0x0a701f (0xf415 bytes) 000317[1] 1788726846.805809 read 0x508000 thru 0x50a5f8 (0x25f9 bytes) 000318[1] 1788726846.827742 punch from 0x6e97d1 to 0x6f8c8b (0xf4bb bytes) 000319[1] 1788726846.832485 read 0x909000 thru 0x90dc22 (0x4c23 bytes) 000320[1] 1788726846.841100 write 0x0b4000 thru 0x0bec32 (0xac33 bytes) 000321[1] 1788726846.847614 write 0x5ed000 thru 0x5f3f56 (0x6f57 bytes) 000322[1] 1788726846.853468 read 0x550000 thru 0x554143 (0x4144 bytes) 000323[1] 1788726846.897477 mapwrite 0x976000 thru 0x97a5f4 (0x45f5 bytes) 000324[0] 1788726847.066876 mapwrite 0x73f000 thru 0x74db79 (0xeb7a bytes) 000325[0] 1788726847.278533 read_OD 0x27d000 thru 0x27ffff (0x3000 bytes) 000329[1] 1788726847.311990 write 0x96a000 thru 0x979f0b (0xff0c bytes) 000331[0] 1788726847.333618 write_OD 0x7f4000 thru 0x7fbfff (0x8000 bytes) 000332[1] 1788726847.413276 mapread 0x82a000 thru 0x82b7a9 (0x17aa bytes) 000333[1] 1788726847.427984 mapread 0x52a000 thru 0x536433 (0xc434 bytes) 000335[1] 1788726847.457009 read 0x7ac000 thru 0x7bbc8c (0xfc8d bytes) 000337[0] 1788726847.468169 punch from 0x49adaf to 0x4a6221 (0xb473 bytes) 000340[1] 1788726847.533889 punch from 0x94d14a to 0x94fa91 (0x2948 bytes) 000344[1] 1788726847.631743 mapread 0x82a000 thru 0x82bec6 (0x1ec7 bytes) 000347[0] 1788726847.732255 punch from 0x90b1ed to 0x91178b (0x659f bytes) 000349[0] 1788726847.776707 write_OD 0x661000 thru 0x663fff (0x3000 bytes) 000353[0] 1788726847.842860 write_OD 0x2dd000 thru 0x2eafff (0xe000 bytes) 000354[1] 1788726847.902355 mapwrite 0x023000 thru 0x02601c (0x301d bytes) 000357[0] 1788726848.032968 write_OD 0x940000 thru 0x94afff (0xb000 bytes) 000358[1] 1788726848.100618 read 0x081000 thru 0x0827da (0x17db bytes) 000361[0] 1788726848.111702 trunc from 0x97a5f5 to 0x9a201f (0x27a2b bytes) 000363[1] 1788726848.175322 mapwrite 0x994000 thru 0x995fd1 (0x1fd2 bytes) 000364[1] 1788726848.278752 mapread 0x2b0000 thru 0x2bf8ee (0xf8ef bytes) 000365[0] 1788726848.391348 punch from 0x62e424 to 0x637185 (0x8d62 bytes) 000366[0] 1788726848.421798 mapwrite 0x981000 thru 0x98ab1e (0x9b1f bytes) 000367[0] 1788726848.525328 mapwrite 0x0e7000 thru 0x0eea56 (0x7a57 bytes) 000368[1] 1788726848.596750 punch from 0x552558 to 0x555274 (0x2d1d bytes) 000369[1] 1788726848.628918 mapread 0x574000 thru 0x57d892 (0x9893 bytes) 000370[1] 1788726848.678030 mapread 0x341000 thru 0x346103 (0x5104 bytes) 000373[0] 1788726848.710460 write_OD 0x934000 thru 0x942fff (0xf000 bytes) 000374[0] 1788726848.818330 trunc from 0x9a2020 to 0x33e8ff (0x663720 bytes) 000375[0] 1788726848.910312 trunc from 0x33e900 to 0x5e0d48 (0x2a2449 bytes) 000376[1] 1788726848.962791 punch from 0x092614 to 0x09e378 (0xbd65 bytes) 000377[0] 1788726849.039993 read_OD 0x597000 thru 0x5a4fff (0xe000 bytes) 000384[0] 1788726849.119105 write_OD 0x719000 thru 0x71bfff (0x3000 bytes) 000386[1] 1788726849.190197 punch from 0x6ed204 to 0x6fb939 (0xe736 bytes) 000387[0] 1788726849.217645 mapwrite 0x3e0000 thru 0x3e74f3 (0x74f4 bytes) 000388[0] 1788726849.392413 mapwrite 0x3a3000 thru 0x3addc8 (0xadc9 bytes) 000389[0] 1788726849.508095 punch from 0x221572 to 0x229ec6 (0x8955 bytes) 000390[0] 1788726849.546131 write_OD 0x770000 thru 0x779fff (0xa000 bytes) 000392[0] 1788726849.625479 trunc from 0x77a000 to 0x784d10 (0xad11 bytes) 000393[1] 1788726849.742753 read 0x1fb000 thru 0x1ff28d (0x428e bytes) 000395[1] 1788726849.799202 write 0x5d6000 thru 0x5e4c5e (0xec5f bytes) 000396[1] 1788726849.837018 mapread 0x2ea000 thru 0x2ee550 (0x4551 bytes) 000398[0] 1788726849.866973 write_OD 0x752000 thru 0x753fff (0x2000 bytes) 000400[1] 1788726849.934059 mapread 0x617000 thru 0x625f85 (0xef86 bytes) 000401[0] 1788726849.964113 mapread 0x24c000 thru 0x25794c (0xb94d bytes) 000402[1] 1788726849.988401 write 0x297000 thru 0x2a3fd0 (0xcfd1 bytes) 000403[1] 1788726850.013648 mapread 0x637000 thru 0x638da5 (0x1da6 bytes) 000405[0] 1788726850.043287 trunc from 0x784d11 to 0x12a591 (0x65a77f bytes) 000408[0] 1788726850.240316 read_OD 0x07e000 thru 0x087fff (0xa000 bytes) 000409[1] 1788726850.267236 punch from 0x06575b to 0x07114c (0xb9f2 bytes) 000410[0] 1788726850.322477 read_OD 0x005000 thru 0x00efff (0xa000 bytes) 000411[0] 1788726850.376038 mapwrite 0x455000 thru 0x457611 (0x2612 bytes) 000413[1] 1788726850.647505 write 0x628000 thru 0x630380 (0x8381 bytes) 000416[0] 1788726850.707044 write_OD 0x183000 thru 0x187fff (0x5000 bytes) 000417[1] 1788726850.789037 mapread 0x4d3000 thru 0x4db826 (0x8827 bytes) 000418[0] 1788726850.818717 mapwrite 0x4f2000 thru 0x4f31d5 (0x11d6 bytes) 000421[1] 1788726850.977076 punch from 0x0a79d0 to 0x0b7667 (0xfc98 bytes) 000422[0] 1788726851.038987 mapread 0x291000 thru 0x296d28 (0x5d29 bytes) 000423[1] 1788726851.135744 mapwrite 0x5b2000 thru 0x5be292 (0xc293 bytes) 000424[0] 1788726851.234682 mapread 0x560000 thru 0x56fab7 (0xfab8 bytes) 000426[1] 1788726851.403174 mapread 0x622000 thru 0x62e1cf (0xc1d0 bytes) 000427[0] 1788726851.455044 punch from 0x0233a5 to 0x02e3ca (0xb026 bytes) 000428[1] 1788726851.518863 mapwrite 0x7ac000 thru 0x7b6a36 (0xaa37 bytes) 000429[1] 1788726851.677015 write 0x59d000 thru 0x5abde5 (0xede6 bytes) 000432[1] 1788726851.710925 mapwrite 0x84f000 thru 0x85408c (0x508d bytes) 000434[0] 1788726851.852239 punch from 0x01034f to 0x012a94 (0x2746 bytes) 000435[0] 1788726851.933866 trunc from 0x85408d to 0x31f1c7 (0x534ec5 bytes) 000437[0] 1788726852.008951 read_OD 0x0bf000 thru 0x0cdfff (0xf000 bytes) 000439[1] 1788726852.047280 mapread 0x056000 thru 0x060818 (0xa819 bytes) 000440[1] 1788726852.093562 mapread 0x221000 thru 0x22fc8b (0xec8c bytes) 000442[1] 1788726852.130028 trunc from 0x31f1c8 to 0x412542 (0xf337b bytes) 000444[1] 1788726852.225128 punch from 0x0eca1f to 0x0f8e1a (0xc3fc bytes) 000450[0] 1788726852.391229 write_OD 0x373000 thru 0x37afff (0x8000 bytes) 000451[1] 1788726852.499130 punch from 0x10867b to 0x11305d (0xa9e3 bytes) 000452[0] 1788726852.525649 read_OD 0x352000 thru 0x35dfff (0xc000 bytes) 000454[1] 1788726852.558472 read 0x1c7000 thru 0x1d310d (0xc10e bytes) 000458[1] 1788726852.573337 read 0x37c000 thru 0x381993 (0x5994 bytes) 000459[1] 1788726852.583554 mapread 0x309000 thru 0x31002d (0x702e bytes) 000461[0] 1788726852.601836 write_OD 0x862000 thru 0x868fff (0x7000 bytes) 000462[0] 1788726852.700835 read_OD 0x39f000 thru 0x39ffff (0x1000 bytes) 000464[0] 1788726852.712766 write_OD 0x6f0000 thru 0x6f7fff (0x8000 bytes) 000466[1] 1788726852.772264 trunc from 0x869000 to 0x45dfd7 (0x40b028 bytes) 000467[1] 1788726852.822055 read 0x266000 thru 0x26cb36 (0x6b37 bytes) 000468[0] 1788726852.853322 mapread 0x296000 thru 0x29da1e (0x7a1f bytes) 000470[1] 1788726852.903890 trunc from 0x45dfd8 to 0x171874 (0x2ec763 bytes) 000471[0] 1788726852.973074 mapread 0x055000 thru 0x057756 (0x2757 bytes) 000472[1] 1788726853.019077 write 0x6e9000 thru 0x6eb185 (0x2186 bytes) 000473[0] 1788726853.055755 write_OD 0x431000 thru 0x43cfff (0xc000 bytes) 000475[1] 1788726853.136763 punch from 0x3f0e52 to 0x3ff8fd (0xeaac bytes) 000476[0] 1788726853.172993 mapread 0x69f000 thru 0x6a817a (0x917b bytes) 000478[0] 1788726853.223188 mapread 0x314000 thru 0x320beb (0xcbec bytes) 000481[0] 1788726853.269168 punch from 0x1b79b4 to 0x1c5c98 (0xe2e5 bytes) 000482[0] 1788726853.309249 write_OD 0x5e2000 thru 0x5e7fff (0x6000 bytes) 000484[0] 1788726853.373299 mapread 0x598000 thru 0x59d374 (0x5375 bytes) 000489[1] 1788726853.399337 mapwrite 0x794000 thru 0x7a0670 (0xc671 bytes) 000490[0] 1788726853.574320 mapread 0x4f5000 thru 0x4f530c (0x30d bytes) 000491[0] 1788726853.636831 read_OD 0x200000 thru 0x20efff (0xf000 bytes) 000493[1] 1788726853.653109 write 0x702000 thru 0x71139b (0xf39c bytes) 000494[0] 1788726853.678095 mapread 0x0f7000 thru 0x106e81 (0xfe82 bytes) 000495[1] 1788726853.761800 punch from 0x14f1d4 to 0x153ef2 (0x4d1f bytes) 000496[0] 1788726853.790132 mapwrite 0x1e6000 thru 0x1ee3ee (0x83ef bytes) 000497[0] 1788726853.968568 mapread 0x52d000 thru 0x5322cb (0x52cc bytes) All operations completed A-OK! PASS 16a (66s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 16:34:20 (1788726860) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.337356 s, 24.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.282606 s, 29.7 MB/s Chance of close/open is 1 in 50 Seed set to 6863 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero length punch hole skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x86f84c 000014[0] 1788726864.544039 trunc from 00000000 to 0x86f84b (0x86f84c bytes) 000015[0] 1788726864.582197 read 0x0cdec2 thru 0x0cf98d (0x1acc bytes) 000016[0] 1788726864.590211 mapread 0x7f7230 thru 0x7f8f58 (0x1d29 bytes) 000017[0] 1788726864.608227 mapread 0x46814e thru 0x468cad (0xb60 bytes) 000018[1] 1788726864.645144 punch from 0x76d60d to 0x76db67 (0x55b bytes) 000019[1] 1788726864.679255 mapwrite 0x7970ab thru 0x7979c2 (0x918 bytes) 000022[1] 1788726864.886029 trunc from 0x86f84c to 0x5a2b98 (0x2cccb3 bytes) 000025[1] 1788726865.031685 punch from 0x2ff19a to 0x30082c (0x1693 bytes) 000026[1] 1788726865.077875 punch from 0x4ef4d9 to 0x4efaca (0x5f2 bytes) 000027[0] 1788726865.083555 mapread 0x2402d3 thru 0x241725 (0x1453 bytes) 000030[1] 1788726865.167746 punch from 0x0eebc4 to 0x0efe29 (0x1266 bytes) 000031[1] 1788726865.207798 mapwrite 0x8993db thru 0x89a79c (0x13c2 bytes) 000032[0] 1788726865.370862 read 0x12da2e thru 0x12f53b (0x1b0e bytes) 000034[1] 1788726865.405677 write 0x913313 thru 0x914be2 (0x18d0 bytes) 000037[1] 1788726865.474740 mapwrite 0x89ac8d thru 0x89b988 (0xcfc bytes) 000038[0] 1788726865.567619 read 0x077476 thru 0x078aba (0x1645 bytes) 000042[1] 1788726865.615251 read 0x747c71 thru 0x7481eb (0x57b bytes) 000045[0] 1788726865.677347 read 0x506216 thru 0x507f00 (0x1ceb bytes) 000048[0] 1788726865.729182 trunc from 0x914be3 to 0x4e2ef3 (0x431cef bytes) 000049[1] 1788726865.803604 write 0x8cc492 thru 0x8cd7a6 (0x1315 bytes) 000051[1] 1788726865.881991 mapread 0x45e8b8 thru 0x45eba2 (0x2eb bytes) 000052[1] 1788726865.935886 read 0x625f57 thru 0x626552 (0x5fc bytes) 000053[0] 1788726865.948987 read 0x37e4a2 thru 0x37e527 (0x86 bytes) 000055[1] 1788726865.981198 read 0x52ad3b thru 0x52b24d (0x513 bytes) 000056[0] 1788726866.001025 punch from 0x1aba30 to 0x1acfd0 (0x15a1 bytes) 000057[1] 1788726866.039041 trunc from 0x8cd7a7 to 0x3b9d91 (0x513a15 bytes) 000058[0] 1788726866.100684 mapwrite 0x028635 thru 0x029f45 (0x1911 bytes) 000059[0] 1788726866.233089 read 0x379c3f thru 0x37a130 (0x4f2 bytes) 000060[1] 1788726866.344772 mapwrite 0x77f830 thru 0x7812b1 (0x1a82 bytes) 000061[1] 1788726866.527221 read 0x0cfa74 thru 0x0d052d (0xaba bytes) 000064[0] 1788726866.641212 punch from 0x61a391 to 0x61a6b6 (0x326 bytes) 000065[0] 1788726866.695640 write 0x58ee55 thru 0x58f5de (0x78a bytes) 000066[0] 1788726866.731843 punch from 0x769d41 to 0x76bc9f (0x1f5f bytes) 000068[0] 1788726866.811090 trunc from 0x7812b2 to 0x392a28 (0x3ee889 bytes) 000069[1] 1788726866.846657 read 0x0c1e0a thru 0x0c3369 (0x1560 bytes) 000075[1] 1788726866.906465 trunc from 0x392a29 to 0x173462 (0x21f5c6 bytes) 000077[1] 1788726867.081727 mapwrite 0x1b2dbb thru 0x1b341c (0x662 bytes) 000078[1] 1788726867.280353 write 0x92d2b9 thru 0x92e48d (0x11d5 bytes) 000079[1] 1788726867.317719 mapwrite 0x8de356 thru 0x8df1ef (0xe9a bytes) 000082[1] 1788726867.431796 mapwrite 0x7301b9 thru 0x73214c (0x1f94 bytes) truncating to largest ever: 0x917120 000085[0] 1788726867.534824 trunc from 0x92e48e to 0x91711f (0x1736e bytes) 000090[0] 1788726867.677170 punch from 0x614f75 to 0x615c77 (0xd03 bytes) 000091[1] 1788726867.688052 mapread 0x030c3d thru 0x031f86 (0x134a bytes) 000093[1] 1788726867.787681 read 0x2f6fa6 thru 0x2f747f (0x4da bytes) 000094[1] 1788726867.814636 mapread 0x82dbff thru 0x82e4f4 (0x8f6 bytes) 000096[0] 1788726867.873778 mapwrite 0x42f852 thru 0x43121a (0x19c9 bytes) 000099[0] 1788726868.047200 mapwrite 0x4337d5 thru 0x43488b (0x10b7 bytes) 000100[0] 1788726868.094527 write 0x745998 thru 0x7472e3 (0x194c bytes) truncating to largest ever: 0x9a295e 000102[0] 1788726868.096603 trunc from 0x917120 to 0x9a295d (0x8b83e bytes) 000103[1] 1788726868.163246 punch from 0x09569e to 0x096883 (0x11e6 bytes) 000105[1] 1788726868.219252 mapwrite 0x42beae thru 0x42d2ce (0x1421 bytes) 000107[1] 1788726868.450043 write 0x6d46fa thru 0x6d5f16 (0x181d bytes) 000108[0] 1788726868.506493 mapread 0x555920 thru 0x556c9c (0x137d bytes) 000109[1] 1788726868.563653 punch from 0x33ff08 to 0x341516 (0x160f bytes) 000111[1] 1788726868.587489 read 0x3e0968 thru 0x3e13ad (0xa46 bytes) 000112[0] 1788726868.608781 trunc from 0x9a295e to 0x698fc1 (0x30999c bytes) 000115[0] 1788726868.726755 mapwrite 0x5f4dc4 thru 0x5f5ab2 (0xcef bytes) 000116[0] 1788726868.792816 read 0x36802c thru 0x368ded (0xdc2 bytes) 000117[0] 1788726868.798789 write 0x733cdc thru 0x734913 (0xc38 bytes) 000119[0] 1788726868.823681 trunc from 0x734914 to 0x98a834 (0x255f21 bytes) 000120[0] 1788726868.877848 trunc from 0x98a835 to 0x5c27b6 (0x3c807e bytes) 000121[1] 1788726868.904901 punch from 0x2e4224 to 0x2e5e6b (0x1c48 bytes) 000123[1] 1788726868.964904 read 0x1828e7 thru 0x184567 (0x1c81 bytes) 000124[1] 1788726868.989211 mapwrite 0x28ab22 thru 0x28c0a8 (0x1587 bytes) 000125[1] 1788726869.095325 write 0x027617 thru 0x027a9d (0x487 bytes) 000129[0] 1788726869.191580 mapread 0x1b424b thru 0x1b56aa (0x1460 bytes) 000130[0] 1788726869.240436 read 0x58fd1a thru 0x58fdf5 (0xdc bytes) 000132[1] 1788726869.268921 mapread 0x516a05 thru 0x5173e7 (0x9e3 bytes) 000133[1] 1788726869.329849 trunc from 0x5c27b7 to 0x2d60b8 (0x2ec6fe bytes) 000134[1] 1788726869.404402 write 0x9f821f thru 0x9fa17f (0x1f61 bytes) 000135[0] 1788726869.457765 write 0x061e5e thru 0x062fb0 (0x1153 bytes) 000136[1] 1788726869.533088 mapwrite 0x578392 thru 0x57a1d5 (0x1e44 bytes) 000137[0] 1788726869.769746 write 0x5696ef thru 0x56ac11 (0x1523 bytes) 000138[1] 1788726869.821773 write 0x17f9f0 thru 0x180533 (0xb44 bytes) 000139[0] 1788726869.880810 punch from 0x1f9e66 to 0x1fb47c (0x1617 bytes) 000140[0] 1788726869.949102 mapread 0x8a4747 thru 0x8a4a1a (0x2d4 bytes) 000141[0] 1788726869.987279 write 0x0ab388 thru 0x0abe28 (0xaa1 bytes) 000143[0] 1788726869.993817 trunc from 0x9fa180 to 0x4ab6b1 (0x54eace bytes) 000144[1] 1788726870.041924 read 0x4544e5 thru 0x454a63 (0x57f bytes) 000145[1] 1788726870.081674 mapread 0x2890a4 thru 0x28a7dc (0x1739 bytes) 000146[1] 1788726870.127026 trunc from 0x4ab6b2 to 0x4021ba (0xa94f7 bytes) 000148[0] 1788726870.185717 mapwrite 0x53f18c thru 0x53fe36 (0xcab bytes) 000149[0] 1788726870.312137 punch from 0x0117f8 to 0x01208c (0x895 bytes) 000150[1] 1788726870.374723 punch from 0x32f70d to 0x3311fe (0x1af2 bytes) 000152[1] 1788726870.435923 punch from 0x053b3b to 0x054554 (0xa1a bytes) 000153[0] 1788726870.459084 punch from 0x22f97e to 0x23155a (0x1bdd bytes) 000158[1] 1788726870.613909 mapread 0x01ab82 thru 0x01b904 (0xd83 bytes) 000159[0] 1788726870.685088 punch from 0x2e6966 to 0x2e7910 (0xfab bytes) 000162[0] 1788726870.766304 mapwrite 0x479cd9 thru 0x47b124 (0x144c bytes) 000163[0] 1788726870.821570 read 0x4c8c4a thru 0x4c968d (0xa44 bytes) 000164[0] 1788726870.826957 punch from 0x35bb1c to 0x35c3dd (0x8c2 bytes) 000167[0] 1788726870.930487 mapwrite 0x96db41 thru 0x96f4c6 (0x1986 bytes) 000169[0] 1788726871.048926 read 0x7b01c2 thru 0x7b17a2 (0x15e1 bytes) 000170[1] 1788726871.116858 read 0x7fc005 thru 0x7fcf71 (0xf6d bytes) 000171[0] 1788726871.162591 trunc from 0x96f4c7 to 0x771ce6 (0x1fd7e0 bytes) 000172[0] 1788726871.255117 mapwrite 0x34d0ed thru 0x34eb2a (0x1a3e bytes) 000173[1] 1788726871.324302 punch from 0x603860 to 0x604800 (0xfa1 bytes) 000174[1] 1788726871.364677 mapwrite 0x513eeb thru 0x514bf4 (0xd0a bytes) 000175[0] 1788726871.472904 trunc from 0x771ce7 to 0x989335 (0x21764f bytes) 000176[1] 1788726871.535650 write 0x9fcd1b thru 0x9fd584 (0x86a bytes) 000178[0] 1788726871.606103 punch from 0x19ffe7 to 0x1a1e9a (0x1eb4 bytes) 000179[0] 1788726871.657953 read 0x3a9e80 thru 0x3aaf70 (0x10f1 bytes) 000180[0] 1788726871.674440 read 0x112a74 thru 0x114451 (0x19de bytes) truncating to largest ever: 0x9e755d 000181[0] 1788726871.750279 trunc from 0x9fd585 to 0x9e755c (0x16028 bytes) 000184[1] 1788726871.819132 trunc from 0x9e755d to 0x48950d (0x55e04f bytes) 000187[0] 1788726871.979110 punch from 0x2c61ad to 0x2c6c15 (0xa69 bytes) 000188[0] 1788726872.038784 read 0x2c224e thru 0x2c3d97 (0x1b4a bytes) 000189[1] 1788726872.091226 mapread 0x106fcb thru 0x1077ff (0x835 bytes) 000190[0] 1788726872.171576 write 0x8bd158 thru 0x8be7da (0x1683 bytes) 000191[1] 1788726872.207498 mapwrite 0x97b073 thru 0x97b7e6 (0x774 bytes) 000195[1] 1788726872.383260 write 0x62ee80 thru 0x62f023 (0x1a4 bytes) 000196[0] 1788726872.449232 read 0x937c45 thru 0x937f71 (0x32d bytes) 000198[0] 1788726872.490076 mapread 0x122534 thru 0x1231c2 (0xc8f bytes) 000199[0] 1788726872.526243 mapread 0x7506fb thru 0x751095 (0x99b bytes) 000201[1] 1788726872.573089 mapwrite 0x416f71 thru 0x418ef3 (0x1f83 bytes) 000202[1] 1788726872.668612 write 0x0c8d47 thru 0x0c97f4 (0xaae bytes) 000205[0] 1788726872.738630 read 0x5f2f91 thru 0x5f382f (0x89f bytes) 000207[1] 1788726872.768332 mapread 0x92222f thru 0x923f61 (0x1d33 bytes) 000209[0] 1788726872.820115 mapread 0x4be0d8 thru 0x4be2e0 (0x209 bytes) 000210[0] 1788726872.863244 write 0x12134e thru 0x122c87 (0x193a bytes) 000211[0] 1788726872.935063 mapwrite 0x2c633e thru 0x2c7349 (0x100c bytes) 000212[1] 1788726873.079947 trunc from 0x97b7e7 to 0x4be3a8 (0x4bd43e bytes) 000214[0] 1788726873.153023 mapread 0x0fc178 thru 0x0fc9e5 (0x86e bytes) 000215[1] 1788726873.221641 punch from 0x0fd40b to 0x0fe79b (0x1391 bytes) 000220[1] 1788726873.438480 read 0x1ed89a thru 0x1ef416 (0x1b7d bytes) 000221[0] 1788726873.448882 mapwrite 0x0ca956 thru 0x0cb913 (0xfbe bytes) 000222[0] 1788726873.637326 trunc from 0x4be3a9 to 0x6c35eb (0x205243 bytes) 000225[0] 1788726873.684739 trunc from 0x6c35ec to 0x2b7406 (0x40c1e5 bytes) 000226[0] 1788726873.723990 read 0x2af152 thru 0x2af74e (0x5fd bytes) 000228[0] 1788726873.838065 mapwrite 0x5a2b27 thru 0x5a37c6 (0xca0 bytes) 000229[1] 1788726874.015780 write 0x0e9d6c thru 0x0eaf70 (0x1205 bytes) 000232[1] 1788726874.098100 mapwrite 0x30faed thru 0x310924 (0xe38 bytes) 000233[1] 1788726874.245199 mapwrite 0x1b8279 thru 0x1b8987 (0x70f bytes) 000238[1] 1788726874.464729 punch from 0x2a3895 to 0x2a39bb (0x127 bytes) 000240[0] 1788726874.518758 trunc from 0x5a37c7 to 0x3119b5 (0x291e11 bytes) 000241[1] 1788726874.590583 mapwrite 0x8a27d6 thru 0x8a3a74 (0x129f bytes) 000242[0] 1788726874.788174 punch from 0x8a2fb5 to 0x8a3a74 (0xac0 bytes) 000245[1] 1788726874.878148 read 0x72b9ca thru 0x72cb7e (0x11b5 bytes) 000246[1] 1788726874.901907 mapwrite 0x0889dc thru 0x089145 (0x76a bytes) 000247[1] 1788726874.989623 mapread 0x587ee6 thru 0x588373 (0x48e bytes) 000248[0] 1788726875.000059 mapwrite 0x8c7d4e thru 0x8c9d35 (0x1fe8 bytes) 000249[1] 1788726875.207023 mapwrite 0x9374bd thru 0x938450 (0xf94 bytes) 000251[0] 1788726875.342451 read 0x33256e thru 0x3336c8 (0x115b bytes) 000253[0] 1788726875.376303 read 0x317d69 thru 0x31890f (0xba7 bytes) 000254[0] 1788726875.399339 mapread 0x17255a thru 0x17296a (0x411 bytes) 000256[0] 1788726875.448057 write 0x5b98d6 thru 0x5b9dcc (0x4f7 bytes) 000257[0] 1788726875.477464 punch from 0x249fc0 to 0x24a72f (0x770 bytes) 000259[0] 1788726875.517561 mapwrite 0x0553f5 thru 0x056136 (0xd42 bytes) 000262[0] 1788726875.588169 mapwrite 0x3ab8f4 thru 0x3ad353 (0x1a60 bytes) 000263[0] 1788726875.671613 trunc from 0x938451 to 0x6eff7c (0x2484d4 bytes) 000264[0] 1788726875.695281 mapwrite 0x238f24 thru 0x23a548 (0x1625 bytes) 000265[0] 1788726875.804212 write 0x291f26 thru 0x293261 (0x133c bytes) 000266[1] 1788726875.818167 read 0x173d49 thru 0x174d0a (0xfc2 bytes) 000267[0] 1788726875.877186 write 0x5eae05 thru 0x5eba46 (0xc42 bytes) 000268[1] 1788726875.904247 mapwrite 0x0674d3 thru 0x067f7f (0xaad bytes) 000270[1] 1788726876.069482 write 0x5ace3d thru 0x5aeab2 (0x1c76 bytes) 000272[1] 1788726876.171725 mapwrite 0x719ba0 thru 0x71a08e (0x4ef bytes) 000274[0] 1788726876.327991 mapwrite 0x88acc1 thru 0x88c582 (0x18c2 bytes) 000276[0] 1788726876.598420 write 0x9056cf thru 0x90731f (0x1c51 bytes) 000277[1] 1788726876.629751 mapwrite 0x45d26a thru 0x45e122 (0xeb9 bytes) 000280[0] 1788726876.882006 read 0x1f17a7 thru 0x1f23fa (0xc54 bytes) 000282[1] 1788726876.956983 mapread 0x1e6075 thru 0x1e6e67 (0xdf3 bytes) 000283[1] 1788726877.032966 read 0x51378f thru 0x515561 (0x1dd3 bytes) 000284[0] 1788726877.065120 write 0x02ed49 thru 0x03014e (0x1406 bytes) 000285[1] 1788726877.165169 mapwrite 0x642ba9 thru 0x643ead (0x1305 bytes) 000286[0] 1788726877.366655 mapread 0x1db9b2 thru 0x1dcfe9 (0x1638 bytes) 000287[1] 1788726877.406243 write 0x875cf6 thru 0x87759c (0x18a7 bytes) 000288[1] 1788726877.428042 write 0x11a41e thru 0x11b3b1 (0xf94 bytes) 000290[0] 1788726877.459580 read 0x7d575f thru 0x7d70a4 (0x1946 bytes) 000291[0] 1788726877.508795 punch from 0x4ca11b to 0x4cb806 (0x16ec bytes) 000292[0] 1788726877.547503 mapwrite 0x9342b6 thru 0x934ac5 (0x810 bytes) 000293[1] 1788726877.671432 trunc from 0x934ac6 to 0x7208c0 (0x214205 bytes) 000295[1] 1788726877.776842 trunc from 0x7208c1 to 0x28397e (0x49cf42 bytes) 000296[0] 1788726877.868214 punch from 0x0ac6ab to 0x0acd74 (0x6ca bytes) 000297[1] 1788726877.970674 trunc from 0x28397f to 0x7ae778 (0x52adfa bytes) 000300[0] 1788726878.085925 read 0x45dfa9 thru 0x45e482 (0x4da bytes) 000301[0] 1788726878.114313 trunc from 0x7ae779 to 0x14c780 (0x661ff8 bytes) 000304[1] 1788726878.210893 write 0x5a6517 thru 0x5a703a (0xb24 bytes) 000305[0] 1788726878.244602 mapread 0x56b3d9 thru 0x56bdf2 (0xa1a bytes) 000307[0] 1788726878.299736 mapwrite 0x0d75de thru 0x0d7a50 (0x473 bytes) 000308[1] 1788726878.507664 mapread 0x17e435 thru 0x17f881 (0x144d bytes) 000310[1] 1788726878.609048 mapread 0x3f84e0 thru 0x3fa445 (0x1f66 bytes) 000314[0] 1788726878.625809 trunc from 0x5a703b to 0x902017 (0x35afdd bytes) 000315[0] 1788726878.702741 mapread 0x8d1a71 thru 0x8d2364 (0x8f4 bytes) 000316[0] 1788726878.720589 write 0x015443 thru 0x0154c5 (0x83 bytes) 000320[1] 1788726878.824493 punch from 0x2b2eba to 0x2b4c35 (0x1d7c bytes) 000321[1] 1788726878.872219 mapwrite 0x7e8d2b thru 0x7ea46c (0x1742 bytes) 000322[1] 1788726878.977324 read 0x59f2e3 thru 0x59f313 (0x31 bytes) 000325[0] 1788726879.077953 punch from 0x5bf3ee to 0x5c02fb (0xf0e bytes) 000327[0] 1788726879.154211 read 0x1378ba thru 0x13954c (0x1c93 bytes) 000329[0] 1788726879.173933 punch from 0x0c7c60 to 0x0c9188 (0x1529 bytes) 000331[0] 1788726879.242309 mapwrite 0x802922 thru 0x802cda (0x3b9 bytes) 000332[1] 1788726879.439073 mapread 0x6b3bc8 thru 0x6b55e3 (0x1a1c bytes) 000334[0] 1788726879.566856 mapwrite 0x311940 thru 0x3126ea (0xdab bytes) 000335[1] 1788726879.668271 trunc from 0x902018 to 0x8b150c (0x50b0b bytes) 000338[0] 1788726879.753732 trunc from 0x8b150d to 0x6ba4f5 (0x1f7017 bytes) 000339[0] 1788726879.842935 trunc from 0x6ba4f6 to 0x50e19d (0x1ac358 bytes) 000340[0] 1788726879.964324 mapwrite 0x5c756d thru 0x5c8e81 (0x1915 bytes) 000344[1] 1788726880.233754 mapread 0x08b355 thru 0x08c0ed (0xd99 bytes) 000346[0] 1788726880.317830 mapwrite 0x43cd5b thru 0x43dd79 (0x101f bytes) 000347[1] 1788726880.461858 read 0x402768 thru 0x403285 (0xb1e bytes) 000348[0] 1788726880.484424 mapread 0x4d1fbe thru 0x4d22e4 (0x327 bytes) 000351[0] 1788726880.507239 write 0x3a8d8c thru 0x3aaaae (0x1d23 bytes) 000353[1] 1788726880.545632 trunc from 0x5c8e82 to 0x9996db (0x3d085a bytes) 000356[0] 1788726880.651084 trunc from 0x9996dc to 0x98ed6d (0xa96e bytes) 000357[0] 1788726880.749902 punch from 0x484e0d to 0x484f9e (0x192 bytes) 000358[1] 1788726880.809637 punch from 0x06f36c to 0x07117c (0x1e11 bytes) 000359[0] 1788726880.933795 mapread 0x8e59b0 thru 0x8e6b24 (0x1175 bytes) 000361[0] 1788726881.016131 mapread 0x685909 thru 0x686196 (0x88e bytes) 000363[1] 1788726881.097000 write 0x27803d thru 0x2787a1 (0x765 bytes) 000366[0] 1788726881.258958 punch from 0x506795 to 0x506c10 (0x47c bytes) 000367[0] 1788726881.303814 write 0x111087 thru 0x1124e3 (0x145d bytes) 000372[0] 1788726881.474421 mapwrite 0x79bf96 thru 0x79d8d1 (0x193c bytes) 000373[1] 1788726881.700248 write 0x8d422a thru 0x8d5f10 (0x1ce7 bytes) 000374[1] 1788726881.835924 punch from 0x64ae1a to 0x64b8bb (0xaa2 bytes) 000376[0] 1788726881.885473 mapread 0x1a55af thru 0x1a57e1 (0x233 bytes) 000377[1] 1788726881.974068 trunc from 0x98ed6e to 0x0181f1 (0x976b7c bytes) 000379[1] 1788726882.020188 write 0x81528a thru 0x8155be (0x335 bytes) 000380[1] 1788726882.063192 punch from 0x7d3be9 to 0x7d4bc2 (0xfda bytes) 000381[0] 1788726882.090792 punch from 0x61628c to 0x617ad9 (0x184e bytes) 000382[0] 1788726882.123236 trunc from 0x8155bf to 0x4139cb (0x401bf3 bytes) 000383[0] 1788726882.189946 read 0x1b066f thru 0x1b0f51 (0x8e3 bytes) 000384[1] 1788726882.220628 read 0x2cee5d thru 0x2cfba7 (0xd4b bytes) 000386[1] 1788726882.264878 trunc from 0x4139cc to 0x1d34f4 (0x2404d7 bytes) 000387[1] 1788726882.337424 trunc from 0x1d34f5 to 0x305543 (0x13204f bytes) 000389[1] 1788726882.444792 write 0x7dbc42 thru 0x7ddbf6 (0x1fb5 bytes) 000391[1] 1788726882.463538 read 0x3e3953 thru 0x3e4e4b (0x14f9 bytes) 000394[0] 1788726882.571180 punch from 0x5c6b48 to 0x5c7b6c (0x1025 bytes) 000398[0] 1788726882.792855 punch from 0x38a8ed to 0x38c877 (0x1f8b bytes) 000399[0] 1788726882.847624 punch from 0x348a58 to 0x34a85e (0x1e07 bytes) 000401[1] 1788726882.901951 mapwrite 0x549dd0 thru 0x54a5f4 (0x825 bytes) 000402[1] 1788726883.017310 mapread 0x6e1ca0 thru 0x6e23f4 (0x755 bytes) 000403[1] 1788726883.054212 mapwrite 0x346df1 thru 0x3481ef (0x13ff bytes) 000405[1] 1788726883.132451 mapwrite 0x603ce7 thru 0x604dc5 (0x10df bytes) 000406[1] 1788726883.205887 write 0x1e28b5 thru 0x1e2cb6 (0x402 bytes) 000407[1] 1788726883.212007 read 0x17bfa8 thru 0x17c706 (0x75f bytes) 000408[0] 1788726883.242358 punch from 0x2234ab to 0x224fd3 (0x1b29 bytes) 000410[0] 1788726883.300788 mapwrite 0x55aa53 thru 0x55ac64 (0x212 bytes) 000411[0] 1788726883.393906 mapwrite 0x8cf2f7 thru 0x8d0087 (0xd91 bytes) 000413[0] 1788726883.645649 mapread 0x669b7f thru 0x66a006 (0x488 bytes) 000414[1] 1788726883.666553 mapread 0x3eb30b thru 0x3ec255 (0xf4b bytes) 000416[0] 1788726883.717465 punch from 0x32076c to 0x321466 (0xcfb bytes) 000417[1] 1788726883.809117 punch from 0x6b57f1 to 0x6b69a7 (0x11b7 bytes) 000418[0] 1788726883.843984 punch from 0x7bf2cd to 0x7c0988 (0x16bc bytes) 000419[0] 1788726883.861963 mapwrite 0x96af42 thru 0x96bbad (0xc6c bytes) 000421[0] 1788726884.060417 write 0x0af3d5 thru 0x0b1331 (0x1f5d bytes) 000423[0] 1788726884.109839 mapwrite 0x4364b6 thru 0x43816f (0x1cba bytes) 000426[0] 1788726884.304191 mapwrite 0x886f9b thru 0x8870f2 (0x158 bytes) 000427[0] 1788726884.401667 write 0x31dd82 thru 0x31f0ef (0x136e bytes) 000428[1] 1788726884.411824 punch from 0x9632cf to 0x964335 (0x1067 bytes) 000430[0] 1788726884.469317 mapread 0x891944 thru 0x89286c (0xf29 bytes) 000432[1] 1788726884.529639 write 0x34f16f thru 0x3505db (0x146d bytes) 000434[0] 1788726884.584797 punch from 0x57b0e3 to 0x57be72 (0xd90 bytes) 000435[0] 1788726884.606886 trunc from 0x96bbae to 0x5d3908 (0x3982a5 bytes) 000438[1] 1788726884.796134 mapread 0x17373d thru 0x173c69 (0x52d bytes) 000439[0] 1788726884.883738 mapread 0x508e86 thru 0x509e9a (0x1015 bytes) 000440[0] 1788726884.921079 write 0x355621 thru 0x355a5f (0x43f bytes) 000441[0] 1788726885.004114 write 0x07bcc3 thru 0x07bd23 (0x61 bytes) 000442[1] 1788726885.040346 punch from 0x0bdace to 0x0bfac7 (0x1ffa bytes) 000445[1] 1788726885.143260 mapwrite 0x8dfaed thru 0x8dff91 (0x4a5 bytes) 000446[0] 1788726885.401818 read 0x38fa5a thru 0x3905c8 (0xb6f bytes) 000447[0] 1788726885.445225 read 0x408c24 thru 0x409405 (0x7e2 bytes) 000448[1] 1788726885.479272 read 0x7482fc thru 0x748388 (0x8d bytes) 000449[1] 1788726885.487514 write 0x3f1735 thru 0x3f302f (0x18fb bytes) 000451[1] 1788726885.537899 mapread 0x883652 thru 0x88371b (0xca bytes) 000452[0] 1788726885.556235 punch from 0x2d78ef to 0x2d9172 (0x1884 bytes) 000453[1] 1788726885.589602 read 0x44334d thru 0x443f91 (0xc45 bytes) 000454[1] 1788726885.606533 punch from 0x689358 to 0x68b1be (0x1e67 bytes) skipping zero size read 000457[1] 1788726885.676219 trunc from 0x8dff92 to 0x7f036b (0xefc26 bytes) 000459[0] 1788726885.772089 read 0x41fbd1 thru 0x420031 (0x461 bytes) 000464[0] 1788726885.878337 write 0x235451 thru 0x235df9 (0x9a9 bytes) 000466[0] 1788726885.920302 trunc from 0x7f036c to 0x0ed795 (0x702bd6 bytes) 000467[1] 1788726885.970308 read 0x0b169b thru 0x0b3310 (0x1c76 bytes) 000468[0] 1788726886.037220 write 0x22ee26 thru 0x22fb75 (0xd50 bytes) 000470[1] 1788726886.051694 write 0x1459ff thru 0x1473c2 (0x19c4 bytes) 000471[1] 1788726886.105959 mapread 0x0845cc thru 0x084db9 (0x7ee bytes) 000477[1] 1788726886.217417 mapwrite 0x8d531f thru 0x8d5f3d (0xc1f bytes) 000479[0] 1788726886.390942 punch from 0x3d0202 to 0x3d18f3 (0x16f2 bytes) 000482[0] 1788726886.460858 read 0x8ced57 thru 0x8cfc5c (0xf06 bytes) truncating to largest ever: 0x9ee69b 000483[1] 1788726886.489799 trunc from 0x8d5f3e to 0x9ee69a (0x11875d bytes) 000484[1] 1788726886.543292 read 0x1cb5e5 thru 0x1cbfb6 (0x9d2 bytes) 000485[1] 1788726886.548664 punch from 0x820c37 to 0x821db8 (0x1182 bytes) 000487[1] 1788726886.595545 write 0x68fdb9 thru 0x6913e3 (0x162b bytes) 000488[1] 1788726886.619246 punch from 0x2f34b2 to 0x2f41c9 (0xd18 bytes) 000493[1] 1788726886.674959 read 0x796797 thru 0x7977bc (0x1026 bytes) 000494[0] 1788726886.721918 punch from 0x9a09d9 to 0x9a292d (0x1f55 bytes) 000495[0] 1788726886.772822 read 0x86e274 thru 0x86f6dd (0x146a bytes) 000497[0] 1788726886.848997 write 0x0d88b5 thru 0x0d9a67 (0x11b3 bytes) 000498[0] 1788726886.885023 read 0x0a3deb thru 0x0a5aa5 (0x1cbb bytes) 000500[0] 1788726886.938156 trunc from 0x9ee69b to 0x00a6dc (0x9e3fbe bytes) All operations completed A-OK! PASS 16b (32s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 16:34:52 (1788726892) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 16:34:56 (1788726896) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.541567 s, 15.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.650933 s, 12.9 MB/s ldlm.namespaces.MGC192.168.204.150@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff90c144a07800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff90c15190c800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff90c144a07800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff90c144a07800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.0668 s, 8.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.15103 s, 25.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.08585 s, 34.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.52815 s, 13.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9002 s, 9.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.72581 s, 28.1 MB/s PASS 16d (59s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 16:35:55 (1788726955) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.39071 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147575 s, 7.1 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 16:36:03 (1788726963) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 16:36:32 (1788726992) TIMEOUT PASS 16g (30s) == sanityn test 16h: mmap read after truncate file ======= 16:37:02 (1788727022) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 16:37:10 (1788727030) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 16:37:18 (1788727038) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.13696 s, 12.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.30761 s, 20.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.2519 s, 8.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.33371 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.06502 s, 8.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.8292 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.97198 s, 6.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.17556 s, 12.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.4674 s, 10.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.60906 s, 7.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.90271 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.44936 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.6233 s, 10.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.39133 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.39795 s, 7.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.54954 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.60186 s, 5.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.69 s, 5.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.91706 s, 5.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (46s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 16:38:05 (1788727085) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Sun Sep 6 16:39:03 EDT 2026 All operations completed A-OK! + fsx 2 done: rc=0 Sun Sep 6 16:39:03 EDT 2026 + fsx 3 done: rc=0 Sun Sep 6 16:39:03 EDT 2026 + fsx 4 done: rc=0 Sun Sep 6 16:39:03 EDT 2026 All operations completed A-OK! + fsx 5 done: rc=0 Sun Sep 6 16:39:03 EDT 2026 PASS 16k (70s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 16:39:15 (1788727155) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 16:39:28 (1788727168) mmap test1: basic mmap operation (PASS, 0.19034s) mmap test2: MAP_PRIVATE not write back (PASS, 0.11872s) mmap test3: concurrent mmap ops on two nodes (PASS, 12.621s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7352s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.27003s) mmap test6: check mmap write/read content on two nodes (PASS, 0.32593s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.354s) mmap test11: multi-thread shared mmap (PASS, 1.5521s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1502s) PASS 18 (32s) == sanityn test 19: test concurrent uncached read races ========================================================================= 16:40:00 (1788727200) oleg450-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 16:40:05 (1788727205) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 16:40:14 (1788727214) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 16:40:22 (1788727222) now is 1788727225 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 new atime is 1788727286 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 16:41:34 (1788727294) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 16:41:42 (1788727302) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 16:41:49 (1788727309) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 16:41:57 (1788727317) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 16:42:00 (1788727320) utime: good mknod times 1788727321 <= 1788727322 <= 1788727322 for /mnt/lustre/f26a utime: good mknod times 1788727321 <= 1788727322 <= 1788727322 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 16:42:08 (1788727328) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 16:42:18 (1788727338) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-06 16:42:20.000000000 -0400 Birth: 2026-09-06 16:42:20.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-06 16:42:20.000000000 -0400 Birth: 2026-09-06 16:42:20.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 16:42:28 (1788727348) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 16:42:37 (1788727357) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 16:42:46 (1788727366) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.150@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (23s) == sanityn test 31r: open-rename(replace) race =========== 16:43:10 (1788727390) -rw-r--r-- 1 root root 0 Sep 6 16:43 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 16:43:20 (1788727400) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43184 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 16:43:29 (1788727409) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 16:43:37 (1788727417) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 16:43:42 (1788727422) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.8974 s, 27.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.57751 s, 33.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.0908 s, 25.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.40858 s, 21.8 MB/s contention_seconds 4, contention_events 19 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.2893 s, 22.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.53435 s, 20.7 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.23046 s, 42.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33745 s, 39.2 MB/s PASS 32c (34s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 16:44:17 (1788727457) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 16:44:21 (1788727461) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 16:44:25 (1788727465) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 16:44:29 (1788727469) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 16:44:32 (1788727472) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90c144a07800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90c144a07800.ost_server_uuid in IDLE state after 0 sec oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c144a07800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90c144a07800.ost_server_uuid in FULL state after 0 sec oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90c144a07800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90c144a07800.ost_server_uuid in IDLE state after 0 sec oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c144a07800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90c144a07800.ost_server_uuid in FULL state after 0 sec oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90c144a07800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90c144a07800.ost_server_uuid in IDLE state after 0 sec oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90c144a07800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff90c144a07800.ost_server_uuid in FULL state after 0 sec PASS 34 (143s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 16:46:56 (1788727616) fail_loc=0x80000317 Race attempt 0 Wait for 50485 50514 for 60 sec... fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.150@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.150@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 640 (time 1788727633.05 total 10.01 last 63.95) - open/close 1303 (time 1788727643.06 total 20.02 last 66.22) - open/close 1943 (time 1788727653.07 total 30.03 last 63.92) - open/close 2545 (time 1788727663.08 total 40.04 last 60.17) - open/close 3149 (time 1788727673.10 total 50.06 last 60.26) - open/close 3778 (time 1788727683.12 total 60.08 last 62.80) total: 4000 open/close in 65.01 seconds: 61.53 ops/second PASS 35 (74s) == sanityn test 36: handle ESTALE/open-unlink correctly == 16:48:09 (1788727689) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg450-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.842857 s, 62.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7427072) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.759469 s, 69.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.695435 s, 75.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7488512) after_dd(7437312) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.799326 s, 65.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7492608) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.620901 s, 84.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7488512) after_dd(7439360) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.662844 s, 79.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7434240) after(7485440) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.655746 s, 80.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7485440) after_dd(7432192) after(7483392) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.665628 s, 78.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7483392) after_dd(7425024) after(7476224) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.867845 s, 60.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7476224) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.718684 s, 73.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7488512) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.543987 s, 96.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7488512) after_dd(7437312) after(7488512) PASS 36 (295s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 16:53:04 (1788727984) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 - create 1284 (time 1788727998.51 total 10.00 last 128.34) - create 2554 (time 1788728008.51 total 20.01 last 126.97) - create 3755 (time 1788728018.51 total 30.01 last 120.10) - create 4952 (time 1788728028.52 total 40.01 last 119.61) - create 6170 (time 1788728038.52 total 50.02 last 121.78) - create 7290 (time 1788728048.52 total 60.02 last 111.97) - create 8467 (time 1788728058.52 total 70.02 last 117.69) - create 9657 (time 1788728068.52 total 80.02 last 118.96) total: 10000 create in 82.81 seconds: 120.76 ops/second PASS 37 (94s) == sanityn test 39a: file mtime does not change after rename ========================================================== 16:54:38 (1788728078) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 16:54:45 (1788728085) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 16:54:54 (1788728094) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 16:55:01 (1788728101) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 16:55:10 (1788728110) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 16:55:29 (1788728129) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 16:55:46 (1788728146) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (14s) == sanityn test 40e: pdirops: rename and others ======================================================================== 16:56:00 (1788728160) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (15s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 16:56:15 (1788728175) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (11s) == sanityn test 41b: pdirops: create vs create ======================================================================== 16:56:26 (1788728186) open: File exists Conflict PASS 41b (11s) == sanityn test 41c: pdirops: create vs link ======================================================================== 16:56:37 (1788728197) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (11s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 16:56:48 (1788728208) Conflict PASS 41d (11s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 16:56:59 (1788728219) Conflict PASS 41e (12s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 16:57:11 (1788728231) Conflict PASS 41f (11s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 16:57:22 (1788728242) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 16:57:34 (1788728254) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (11s) == sanityn test 41i: reint_open: create vs create ======== 16:57:45 (1788728265) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (979s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 17:14:04 (1788729244) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (10s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 17:14:14 (1788729254) open: File exists Conflict PASS 42b (10s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 17:14:24 (1788729264) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (11s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 17:14:35 (1788729275) Conflict PASS 42d (10s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 17:14:45 (1788729285) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (9s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 17:14:55 (1788729295) Conflict PASS 42f (11s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 17:15:05 (1788729305) Conflict PASS 42g (10s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 17:15:15 (1788729315) Conflict PASS 42h (10s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 17:15:25 (1788729325) PASS 43a (53s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 17:16:18 (1788729378) Conflict PASS 43b (9s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 17:16:27 (1788729387) Conflict PASS 43c (10s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 17:16:37 (1788729397) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (10s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 17:16:47 (1788729407) Conflict PASS 43e (11s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 17:16:58 (1788729418) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (11s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 17:17:10 (1788729430) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (10s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 17:17:20 (1788729440) Conflict PASS 43h (10s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 17:17:30 (1788729450) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 17:17:33 (1788729453) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (92s) == sanityn test 43k: unlink vs create ==================== 17:19:05 (1788729545) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (749s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 17:31:34 (1788730294) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun Sep 6 17:31:39 EDT 2026 PASS 44a (7s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 17:31:41 (1788730301) open: File exists Conflict PASS 44b (8s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 17:31:49 (1788730309) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (7s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 17:31:56 (1788730316) Conflict PASS 44d (7s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 17:32:03 (1788730323) Conflict PASS 44e (7s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 17:32:10 (1788730330) Conflict PASS 44f (7s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 17:32:17 (1788730337) Conflict PASS 44g (7s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 17:32:24 (1788730344) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 17:32:30 (1788730350) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 17:32:32 (1788730352) PASS 45a (49s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 17:33:21 (1788730401) Conflict PASS 45b (6s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 17:33:27 (1788730407) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 17:33:33 (1788730413) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (6s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 17:33:39 (1788730419) Conflict PASS 45e (6s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 17:33:45 (1788730425) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 17:33:51 (1788730431) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (6s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 17:33:57 (1788730437) Conflict PASS 45h (6s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 17:34:03 (1788730443) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 17:34:04 (1788730444) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (496s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 17:42:21 (1788730941) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 17:42:26 (1788730946) open: File exists Conflict PASS 46b (6s) == sanityn test 46c: pdirops: link vs link ======================================================================== 17:42:32 (1788730952) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 17:42:37 (1788730957) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 17:42:43 (1788730963) Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 17:42:49 (1788730969) Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 17:42:55 (1788730975) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 17:43:00 (1788730980) Conflict PASS 46h (6s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 17:43:06 (1788730986) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 17:43:08 (1788730988) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 17:43:09 (1788730989) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 17:43:10 (1788730990) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 17:43:12 (1788730992) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 17:43:13 (1788730993) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 17:43:14 (1788730994) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 17:43:15 (1788730995) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 17:43:16 (1788730996) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00534366 s, 1.9 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 17:43:24 (1788731004) 0+1 records in 0+1 records out 158 bytes copied, 0.00354309 s, 44.6 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218407) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 17:43:28 (1788731008) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00352874 s, 290 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119618 s, 87.7 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 17:43:43 (1788731023) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00612519 s, 167 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 17:43:50 (1788731030) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117957 s, 88.9 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 17:43:54 (1788731034) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 17:43:58 (1788731038) llite.lustre-ffff90c144a07800.xattr_cache=1 llite.lustre-ffff90c15190c800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 17:43:58.000000000 -0400 Modify: 2026-09-06 17:43:58.000000000 -0400 Change: 2026-09-06 17:43:58.000000000 -0400 Birth: 2026-09-06 17:43:58.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 17:44:04.000000000 -0400 Modify: 2026-09-06 17:44:04.000000000 -0400 Change: 2026-09-06 17:44:04.000000000 -0400 Birth: 2026-09-06 17:44:04.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 17:44:09.000000000 -0400 Modify: 2026-09-06 17:44:09.000000000 -0400 Change: 2026-09-06 17:44:09.000000000 -0400 Birth: 2026-09-06 17:44:09.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 17:44:09.000000000 -0400 Modify: 2026-09-06 17:44:10.000000000 -0400 Change: 2026-09-06 17:44:10.000000000 -0400 Birth: 2026-09-06 17:44:09.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 17:44:15.000000000 -0400 Modify: 2026-09-06 17:44:15.000000000 -0400 Change: 2026-09-06 17:44:15.000000000 -0400 Birth: 2026-09-06 17:44:15.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 17:44:15.000000000 -0400 Modify: 2026-09-06 17:44:15.000000000 -0400 Change: 2026-09-06 17:44:15.000000000 -0400 Birth: 2026-09-06 17:44:15.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 17:44:22 (1788731062) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 17:44:30 (1788731070) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 17:44:38 (1788731078) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 17:44:51 (1788731091) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 17:45:01 (1788731101) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 17:45:02 (1788731102) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 17:45:15 (1788731115) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (15s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 17:45:30 (1788731130) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788731130 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.26071 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (29s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 17:46:00 (1788731160) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788731161 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.81482 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (36s) == sanityn test 60: Verify data_version behaviour ======== 17:46:36 (1788731196) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 17:46:39 (1788731199) PASS 70a (2s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 17:46:41 (1788731201) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 17:46:44 (1788731204) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00406615 s, 16.1 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 17:46:46 (1788731206) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00475911 s, 13.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 17:46:47 (1788731207) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 17:46:49 (1788731209) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00460964 s, 14.2 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 17:46:50 (1788731210) llite.lustre-ffff90c144a07800.xattr_cache=1 llite.lustre-ffff90c15190c800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 17:46:52 (1788731212) llite.lustre-ffff90c144a07800.xattr_cache=1 llite.lustre-ffff90c15190c800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 17:46:55 (1788731215) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00247013 s, 415 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00312518 s, 328 kB/s 231180: get lock2 231182: get lock1 231180: done 231180: get lock1 231182: done 231182: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231182: cannot get lock2: Resource deadlock avoided 231182: exit rc=0 231180: done 231180: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 17:47:01 (1788731221) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0119798 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00837902 s, 125 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231763 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 17:47:08 (1788731228) opening files: ........ closing files: ........ PASS 76 (17s) == sanityn test 77a: check FIFO NRS policy =============== 17:47:25 (1788731245) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0689983 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110888 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298007 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308348 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324828 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350144 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038761 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342415 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417792 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396504 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602766 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057263 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529877 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479514 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623146 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602541 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650822 s, 16.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226188 s, 46.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294589 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122711 s, 85.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277476 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165952 s, 63.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172663 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173643 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190043 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195737 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305329 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250029 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233528 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237864 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239559 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243404 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026073 s, 40.2 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 17:47:29 (1788731249) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0930073 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115237 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329192 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280828 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336404 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346934 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321648 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220809 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291333 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438273 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04985 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342582 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266273 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499457 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154453 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142657 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333939 s, 31.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227523 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233786 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191297 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297392 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224336 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214474 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252179 s, 41.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194833 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253205 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259935 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026976 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02315 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224719 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295528 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282873 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275278 s, 38.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0920011 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117945 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297642 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348778 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392926 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454997 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328873 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037115 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408893 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386564 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500912 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570391 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319473 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490606 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518175 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532018 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389171 s, 26.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182099 s, 57.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119184 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279255 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151225 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207964 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220825 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181692 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211763 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195709 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269773 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229123 s, 45.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279938 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00313005 s, 335 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309705 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287645 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288247 s, 36.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 17:47:34 (1788731254) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.139 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145347 s, 72.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134375 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324658 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219958 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140255 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481961 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441949 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341403 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042205 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235259 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428822 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371547 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219033 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563087 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472727 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539955 s, 19.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016978 s, 61.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012275 s, 85.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209153 s, 50.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302073 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287981 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268237 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030301 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226933 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220103 s, 47.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108873 s, 96.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293882 s, 35.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00475396 s, 221 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00479443 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188503 s, 55.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359415 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01722 s, 60.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0737908 s, 227 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119022 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249426 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116541 s, 90.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445119 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423907 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491261 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417907 s, 25.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501488 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500513 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467409 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476974 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286528 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519488 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296114 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548209 s, 19.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558372 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212644 s, 49.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230903 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229331 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239517 s, 43.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153353 s, 68.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317731 s, 33.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00955949 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298507 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155446 s, 67.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165417 s, 63.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251641 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138229 s, 75.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149009 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146963 s, 71.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116834 s, 89.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116735 s, 89.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.107449 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01106 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018248 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188084 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347694 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307265 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403211 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016004 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041337 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294155 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492778 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721833 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607874 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523157 s, 20.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547868 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677101 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720012 s, 14.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254617 s, 41.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285567 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350556 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307595 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365708 s, 28.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221622 s, 47.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354463 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175144 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234358 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370938 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302606 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285953 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262514 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440432 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215758 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377344 s, 27.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 17:47:40 (1788731260) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0753491 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130888 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158424 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303636 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345082 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359989 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391977 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456628 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308386 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045548 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037363 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528398 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281779 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513377 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248245 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472134 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473574 s, 22.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195002 s, 53.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180226 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220939 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171958 s, 61.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00440556 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232645 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021446 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00905275 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207707 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247407 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630028 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167477 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024213 s, 43.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208174 s, 50.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131168 s, 79.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00532581 s, 197 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0696762 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114318 s, 91.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285876 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142952 s, 73.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236373 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405904 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287996 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230491 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413302 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280905 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327662 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547097 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410198 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388958 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455264 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599009 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656706 s, 16.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130157 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219248 s, 47.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212635 s, 49.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149732 s, 70.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135733 s, 77.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151698 s, 69.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00881864 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161798 s, 64.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134178 s, 78.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254071 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598123 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216575 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246058 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00703591 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152147 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628968 s, 167 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.101585 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224417 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187727 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118915 s, 88.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431064 s, 24.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533094 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424863 s, 24.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341768 s, 30.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060317 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399492 s, 26.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082698 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604798 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822163 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741224 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748734 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735884 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06295 s, 16.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209787 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189351 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019316 s, 54.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00952561 s, 110 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144196 s, 72.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204935 s, 51.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276507 s, 37.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247334 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331068 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125577 s, 83.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187919 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188701 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170329 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111019 s, 94.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101015 s, 104 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00577599 s, 182 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 17:47:45 (1788731265) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.50@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0634418 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139728 s, 75.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418988 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429584 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457025 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532613 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443009 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196827 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221933 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577864 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377707 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718037 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625342 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647257 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527073 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449021 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058238 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181078 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162894 s, 64.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176395 s, 59.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174575 s, 60.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187337 s, 56.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220112 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188078 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189686 s, 55.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202761 s, 51.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171642 s, 61.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00380927 s, 275 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026028 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00456844 s, 230 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242429 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120237 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00446905 s, 235 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0943242 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159875 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235585 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221937 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166991 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128194 s, 81.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406176 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225567 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028127 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111766 s, 93.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912887 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110814 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123815 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122244 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127661 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107148 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129368 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200763 s, 52.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108386 s, 96.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233474 s, 44.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184212 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211709 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208465 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191391 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311187 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223353 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291913 s, 35.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410009 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266078 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277675 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278369 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282041 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298697 s, 35.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0820452 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240451 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464992 s, 22.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475883 s, 22.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420382 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051396 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536451 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492336 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049261 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438256 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368656 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557086 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665617 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429348 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375512 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630644 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510241 s, 20.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251143 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237735 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201737 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276101 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017432 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024135 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00666892 s, 157 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182309 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242304 s, 43.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916029 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260383 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107092 s, 97.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353459 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0067609 s, 155 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281387 s, 37.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00774464 s, 135 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0781272 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106249 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328085 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255278 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186986 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335741 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286748 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112428 s, 93.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303914 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367787 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397723 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243471 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482634 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375541 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465647 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466871 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405832 s, 25.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146912 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177762 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202098 s, 51.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148816 s, 70.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183481 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295007 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258933 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180291 s, 58.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250447 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00624819 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201404 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243107 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705936 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226684 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367878 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00492402 s, 213 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 17:47:53 (1788731273) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.160442 s, 105 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025524 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335281 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196178 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405194 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247239 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340967 s, 30.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290717 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348728 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239655 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469766 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513661 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689651 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430645 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463746 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595956 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636622 s, 16.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148344 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128269 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211405 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168778 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188052 s, 55.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217876 s, 48.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218633 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112536 s, 93.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274088 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143837 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00585855 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00704344 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027277 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043536 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00837052 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00709239 s, 148 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0831302 s, 202 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132322 s, 79.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177331 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369146 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383171 s, 27.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402893 s, 26.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339538 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326142 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426837 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279969 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418459 s, 25.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373649 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316264 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398774 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00881536 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319987 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0088567 s, 118 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012464 s, 84.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140109 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013039 s, 80.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192196 s, 54.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223897 s, 46.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192698 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027373 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267162 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303186 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133531 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266847 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145826 s, 71.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00687758 s, 152 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00497099 s, 211 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00526271 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00915771 s, 115 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.100035 s, 168 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292936 s, 35.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303873 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308662 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468354 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040512 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377074 s, 27.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570026 s, 18.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310807 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312904 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343049 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380558 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375036 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458901 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257692 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369101 s, 28.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370698 s, 28.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025484 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00904953 s, 116 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180592 s, 58.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277533 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335538 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405771 s, 258 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369945 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191035 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261009 s, 40.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348842 s, 30.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278928 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308737 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301064 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00381031 s, 275 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274113 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00737294 s, 142 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0755464 s, 222 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112444 s, 93.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275303 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038169 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380373 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377329 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224705 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366928 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034404 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338065 s, 31.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff90c15190c800.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058781 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484425 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373309 s, 28.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651457 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051996 s, 20.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800878 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641236 s, 16.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166709 s, 62.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180147 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243626 s, 43.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033896 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0082349 s, 127 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00879315 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349791 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211932 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308534 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375698 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287534 s, 36.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c15190c800.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188239 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364742 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180907 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115692 s, 90.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00811892 s, 129 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (8s) == sanityn test 77g: Change TBF type directly ============ 17:48:02 (1788731282) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 17:48:06 (1788731286) oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg450-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg450-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg450-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg450-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 17:48:10 (1788731290) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (9s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 17:48:19 (1788731299) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.0706981 s, 237 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185878 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210382 s, 49.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368862 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318081 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039134 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404007 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257196 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346235 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298062 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300364 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562522 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465542 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487554 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127723 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385853 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258793 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145584 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197159 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162019 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180759 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133153 s, 78.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199751 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018389 s, 57.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209075 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242659 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638196 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00872212 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00667093 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121041 s, 86.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317156 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00902014 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100559 s, 104 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98377 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4536 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (51s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 17:49:10 (1788731350) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg450-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg450-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg450-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg450-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23062 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266211 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358826 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451383 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361684 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365596 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399833 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700537 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05596 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167886 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167812 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182912 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193496 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171229 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176634 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191462 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155491 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025142 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252092 s, 41.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266018 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242652 s, 43.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226393 s, 46.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020597 s, 50.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267771 s, 39.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339157 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218857 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286728 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329721 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013529 s, 77.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033452 s, 31.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194033 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00616445 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00614634 s, 171 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7032 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9467 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg450-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg450-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg450-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg450-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21803 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233854 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026208 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278747 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257446 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018644 s, 56.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316875 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212304 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265249 s, 39.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1615 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161522 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159193 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163493 s, 6.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158723 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177623 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170904 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184062 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138992 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189443 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198518 s, 52.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120444 s, 87.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00947529 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241514 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195938 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295075 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193804 s, 54.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331005 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00545232 s, 192 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345105 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125987 s, 83.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00465952 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331743 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00686566 s, 153 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8364 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9507 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (133s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 17:51:24 (1788731484) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg450-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg450-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg450-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg450-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21723 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334394 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032465 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254692 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475116 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441365 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454075 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336001 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393989 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163213 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163204 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18445 s, 5.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163275 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178182 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156413 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177149 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165179 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255146 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251284 s, 41.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288747 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305282 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278636 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111056 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229972 s, 45.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336449 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172403 s, 60.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159842 s, 65.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118137 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330409 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232883 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035334 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00812248 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102105 s, 103 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8314 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9588 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg450-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg450-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg450-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg450-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21937 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269746 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335638 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309765 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026253 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021012 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025644 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391616 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316364 s, 33.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158899 s, 6.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158917 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151567 s, 6.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159743 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185418 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159768 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191082 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182086 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156685 s, 66.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139009 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150907 s, 69.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309533 s, 33.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319841 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317229 s, 33.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288697 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332485 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219936 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318261 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257566 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256215 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00947754 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00431756 s, 243 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369316 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00889284 s, 118 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6775 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9557 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (137s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 17:53:41 (1788731621) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241913 s, 69.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131176 s, 79.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315266 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395964 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374465 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286858 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425237 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234532 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228274 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189284 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546391 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653037 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411285 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585747 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680454 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577497 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536453 s, 19.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0086041 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114162 s, 91.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146136 s, 71.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152636 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389546 s, 269 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552035 s, 190 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160195 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216891 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267667 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011139 s, 94.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205097 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235237 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277098 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205741 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023941 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122119 s, 85.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9273 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7325 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.50@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.50@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.225179 s, 74.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220077 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163286 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469312 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349426 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608666 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418862 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627114 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434799 s, 24.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484237 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553213 s, 19.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585424 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610614 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258208 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27159 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303516 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296744 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218098 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162029 s, 64.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268498 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179824 s, 58.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251701 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030106 s, 34.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257713 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319879 s, 32.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116172 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02998 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394417 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126403 s, 83.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152229 s, 68.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402444 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148912 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00531961 s, 197 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.954 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73555 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.50@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.229212 s, 73.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146944 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335248 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352275 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299503 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487886 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472319 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195618 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360982 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489872 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039743 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326312 s, 32.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548073 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644767 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226062 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530662 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551322 s, 19.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294836 s, 35.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367061 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220998 s, 47.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426416 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442333 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320394 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485703 s, 21.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473297 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257878 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410548 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283699 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048955 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390602 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395473 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385844 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374544 s, 28.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94952 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.89292 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.426216 s, 39.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221664 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144773 s, 72.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020272 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149679 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178481 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291747 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192698 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169888 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620813 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077244 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921174 s, 11.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921641 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361447 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36749 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415463 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430794 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015741 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01698 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167188 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151928 s, 69.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229597 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238396 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198133 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159391 s, 65.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221054 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026349 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150676 s, 69.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251294 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00364694 s, 288 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00798318 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00284685 s, 368 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240778 s, 435 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78922 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73476 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91233 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73089 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.827783 s, 20.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260585 s, 40.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138101 s, 75.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215092 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198101 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157658 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192979 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372159 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215559 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164211 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178224 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167785 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181708 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193735 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184256 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195902 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170126 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00969036 s, 108 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017002 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188755 s, 55.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149237 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162063 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187838 s, 55.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193548 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168892 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00406603 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038865 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207999 s, 50.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366047 s, 286 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00513872 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021605 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00361611 s, 290 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184241 s, 56.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6692 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4279 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.225984 s, 74.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107506 s, 97.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137924 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224062 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267342 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137023 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376623 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347541 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332173 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252512 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369344 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105455 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035609 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364717 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316555 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237091 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118741 s, 88.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127131 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134146 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162745 s, 64.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149144 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200267 s, 52.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178299 s, 58.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163799 s, 64.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103572 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00353513 s, 297 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216574 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214593 s, 48.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00836428 s, 125 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210774 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00236851 s, 443 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00890506 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247108 s, 42.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94296 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73205 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.427103 s, 39.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122076 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210434 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012413 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127689 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110392 s, 95.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135956 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00795153 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020876 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591918 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079786 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725515 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673713 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371228 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361483 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417119 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404069 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166917 s, 62.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00728692 s, 144 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014818 s, 70.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195533 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152393 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182103 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175632 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00640799 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200674 s, 52.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243104 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019889 s, 52.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218894 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245449 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404465 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570775 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00566977 s, 185 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7942 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7314 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77483 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73273 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (359s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 17:59:40 (1788731980) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.247405 s, 67.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145103 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333677 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221986 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361637 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275644 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378079 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223439 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458921 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429457 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618257 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509607 s, 20.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599808 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539982 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685714 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625725 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636073 s, 16.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159008 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183983 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185426 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019761 s, 53.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158374 s, 66.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117109 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200941 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204179 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265406 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0069131 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216682 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213048 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039563 s, 265 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00456835 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397473 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00847402 s, 124 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92806 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73382 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (37s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 18:00:17 (1788732017) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.50@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.50@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.222562 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160279 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309977 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275262 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363216 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413977 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354098 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455373 s, 23.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493151 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427364 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557733 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559637 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495081 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233175 s, 4.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250019 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276036 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276441 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166349 s, 63.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152444 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170817 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196567 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162429 s, 64.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218669 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172818 s, 60.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259064 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174362 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183438 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161187 s, 65.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200435 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213918 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369408 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0075622 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00668867 s, 157 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91179 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72627 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (36s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 18:00:53 (1788732053) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.50@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241741 s, 69.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169682 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324701 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274188 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286953 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307597 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413884 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301477 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346621 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447191 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243954 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465943 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400139 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549486 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627216 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492444 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047805 s, 21.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120022 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401785 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142715 s, 73.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171951 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131736 s, 79.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145218 s, 72.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178152 s, 58.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138864 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180625 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163891 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00302909 s, 346 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373096 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221802 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460363 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343948 s, 305 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366444 s, 286 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92652 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96613 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (32s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 18:01:25 (1788732085) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.443025 s, 37.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226267 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376565 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294839 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487732 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615286 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430933 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458062 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476032 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575565 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534422 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576248 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725874 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315067 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29143 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363111 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337333 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160004 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194759 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142641 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199819 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182494 s, 57.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194721 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200505 s, 52.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013601 s, 77.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189843 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211755 s, 49.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00813534 s, 129 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477768 s, 219 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180313 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011113 s, 94.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00775468 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00501595 s, 209 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78643 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72683 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91162 s, 21.3 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72963 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (80s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 18:02:46 (1788732166) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.826786 s, 20.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175432 s, 59.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177562 s, 59.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157609 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291195 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147115 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278508 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212569 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304769 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155491 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140834 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163685 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156975 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148292 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176698 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179731 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16962 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201741 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018737 s, 56.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212531 s, 49.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384328 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393098 s, 26.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185947 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226253 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273974 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255854 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385228 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230501 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276798 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032152 s, 32.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047233 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261461 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286898 s, 36.5 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6492 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.962 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 18:03:52 (1788732232) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.225005 s, 74.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105614 s, 99.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299175 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314762 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012154 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282992 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340021 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033794 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237764 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419875 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396093 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120648 s, 86.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368815 s, 28.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473317 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187961 s, 55.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01822 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034332 s, 30.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178358 s, 58.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193948 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216207 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243248 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227213 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249005 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237823 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249652 s, 42.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138258 s, 75.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00533209 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178693 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263092 s, 39.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00745119 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198466 s, 52.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00737949 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221429 s, 47.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92875 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73296 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg450-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.427764 s, 39.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190777 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159173 s, 65.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115203 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227372 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244194 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213245 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034519 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234103 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650973 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698185 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08225 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695097 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272465 s, 3.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264153 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318169 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312132 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130505 s, 80.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196825 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219864 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229311 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211363 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368002 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026789 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00970577 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308568 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189752 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270976 s, 38.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248472 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0073017 s, 144 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142356 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165923 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130573 s, 80.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7827 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73187 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76247 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7282 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (118s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 18:05:50 (1788732350) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg450-client.virtnet /mnt/lustre (opts:) Stopping client oleg450-client.virtnet /mnt/lustre opts: Stopping clients: oleg450-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg450-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg450-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg450-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.50,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg450-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.50,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg450-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.50,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69699/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.50,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg450-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: kmemleak disabled oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: detected 4 online CPUs by sysfs oleg450-server: MODOPTS_LIBCFS= oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre Starting client oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre Started clients oleg450-client.virtnet: 192.168.204.150@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Starting client oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Started clients oleg450-client.virtnet: 192.168.204.150@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff90c142e1b800.idle_timeout=debug osc.lustre-OST0000-osc-ffff90c171daf800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90c142e1b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90c171daf800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.239459 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584444 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841241 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968937 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112424 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129722 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117242 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131385 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135925 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153799 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163119 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133408 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137627 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245566 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260299 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290577 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274115 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139371 s, 75.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149965 s, 69.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139951 s, 74.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158777 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244254 s, 42.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171568 s, 61.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210677 s, 49.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164863 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198747 s, 52.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00443319 s, 237 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426706 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027159 s, 38.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265754 s, 39.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250792 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00964586 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00437637 s, 240 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9373 s, 21.2 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96855 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.258815 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657032 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953851 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135622 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136604 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123295 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148939 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154054 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142206 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180578 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157554 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143089 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148787 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239939 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22442 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295615 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281292 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170482 s, 61.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189477 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016333 s, 64.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198453 s, 52.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245405 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175447 s, 59.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228038 s, 46.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0059349 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208184 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260743 s, 40.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027789 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00496633 s, 211 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00987081 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00831519 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128597 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00199521 s, 526 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9393 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73216 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (110s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 18:07:40 (1788732460) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg450-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (7s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 18:07:47 (1788732467) On MGS 192.168.204.150, active = nodemap.active=1 On MGS 192.168.204.150, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.257013 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255964 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041279 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399792 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532443 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058243 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652984 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763657 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762174 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641081 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078718 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913006 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971589 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268875 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251124 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306633 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300301 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314845 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286151 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221734 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325453 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280855 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330569 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259898 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282147 s, 37.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291992 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244265 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025892 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322317 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281347 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327788 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348909 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332094 s, 31.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92033 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96789 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.150, TBF.id = On MGS 192.168.204.150, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.15458 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382658 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329018 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085182 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937415 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967181 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974302 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906301 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947527 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909138 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075498 s, 13.9 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff90c142e1b800.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834598 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858337 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851062 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859784 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920687 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106657 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176384 s, 59.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183857 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248875 s, 42.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c142e1b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff90c142e1b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304093 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226747 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100071 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221426 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409372 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00865472 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316073 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262631 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166517 s, 63.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280672 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197238 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104051 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0030573 s, 343 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.60721 s, 65.2 MB/s Write runtime is 3 s, speed is 33.333333 IOPS write rate (33.333333) should exceed limit (20 * 1) Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.494715 s, 212 MB/s Read runtime is 1 s, speed is 100.000000 IOPS read rate (100.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg450-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg450-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 On MGS 192.168.204.150, active = nodemap.active=1 On MGS 192.168.204.150, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.248418 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197419 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303655 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594169 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291926 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575135 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799911 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777741 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909461 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791553 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820088 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937104 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104042 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234795 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259853 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302094 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298794 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00923841 s, 114 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143741 s, 72.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215527 s, 48.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134771 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234727 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102737 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222447 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174115 s, 60.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00871653 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016549 s, 63.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012084 s, 86.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00646857 s, 162 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0069142 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00702368 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025436 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478457 s, 219 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92923 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73184 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.150, TBF.id = On MGS 192.168.204.150, active = nodemap.active=0 PASS 77kj (117s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 18:09:44 (1788732584) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 18:09:48 (1788732588) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg450-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.75707 s, 279 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg450-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0034 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 18:10:39 (1788732639) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.124244 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608051 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714777 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564389 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797917 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599105 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110926 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130332 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143011 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14413 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156639 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13855 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147802 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179164 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178959 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134323 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174118 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff90c142e1b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff90c142e1b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256619 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233645 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222236 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258278 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271069 s, 38.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216208 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360807 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265997 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288565 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379441 s, 27.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403273 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424901 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339967 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504714 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423671 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375454 s, 27.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg450-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg450-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93994 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9706 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.24259 s, 69.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292219 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174283 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335842 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492275 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541782 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461935 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770785 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560108 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054867 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464928 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844261 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971211 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916252 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105885 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254079 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265694 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224395 s, 46.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171343 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145427 s, 72.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183598 s, 57.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209838 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208981 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173297 s, 60.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022307 s, 47.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181931 s, 57.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253164 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270999 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249543 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00803974 s, 130 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252373 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00533299 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278517 s, 37.6 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg450-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg450-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93375 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96738 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (65s) == sanityn test 77o: Changing rank should not panic ====== 18:11:44 (1788732704) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 18:11:48 (1788732708) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (38s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 18:12:26 (1788732746) mds.MDS.mdt.nrs_policies=tbf oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg450-client: oleg450-server: ssh exited with exit code 36 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg450-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg450-client: oleg450-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 18:12:38 (1788732758) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 276815 kill md_thread xargs PASS 77r (42s) == sanityn test 77s: Check TBF LRU shrinker ============== 18:13:20 (1788732800) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (13s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 18:13:33 (1788732813) oleg450-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg450-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg450-client: oleg450-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 18:13:36 (1788732816) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826936280 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 18:13:37.000000000 -0400 Modify: 2026-09-06 18:13:37.000000000 -0400 Change: 2026-09-06 18:13:37.000000000 -0400 Birth: 2026-09-06 18:13:37.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 18:13:49 (1788732829) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 18:13:50 (1788732830) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 18:13:51 (1788732831) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 18:13:52 (1788732832) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 18:13:53 (1788732833) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 18:13:55 (1788732835) total: 2500 open/close in 5.38 seconds: 464.48 ops/second snapshot_time 1788732858.193521076 secs.nsecs start_time 1788732835.875116306 secs.nsecs elapsed_time 22.318404770 secs.nsecs open 2505 samples [usecs] 79 1959 894933 346751471 close 2504 samples [usecs] 65 1461 248597 28421815 mknod 2501 samples [usecs] 220 1882 794606 276304340 unlink 2500 samples [usecs] 358 20696 1715531 3631815731 mkdir 2 samples [usecs] 680 878 1558 1233284 rmdir 2 samples [usecs] 2355 3396 5751 17078841 rename 2500 samples [usecs] 299 13686 3145316 5782120298 getattr 3760 samples [usecs] 7 7794 103850 161257960 parallel_rename_file 2500 samples [usecs] 299 13686 3145316 5782120298 crossdir_rename 2500 samples [usecs] 299 13686 3145316 5782120298 readpage 23 samples [usecs] 2055 16711 140715 1122509963 PASS 81d (24s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 18:14:19 (1788732859) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7630 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7630 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 18:14:22 (1788732862) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 18:14:23 (1788732863) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 18:14:30 (1788732870) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788732871 Finishing test test1 at 1788732871 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 18:14:33 (1788732873) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 18:14:34 (1788732874) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 18:14:35 (1788732875) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 18:14:36 (1788732876) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6433 0x1921 0x240000400 1 6395 0x18fb 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6396 0x18fc 0x280000400 0 6434 0x1922 0x240000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 18:14:45 (1788732885) osc.lustre-OST0000-osc-ffff90c142e1b800.idle_timeout=0 osc.lustre-OST0000-osc-ffff90c171daf800.idle_timeout=0 osc.lustre-OST0001-osc-ffff90c142e1b800.idle_timeout=0 osc.lustre-OST0001-osc-ffff90c171daf800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0488851 s, 168 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09673 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 297450 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0137347 s, 298 kB/s osc.lustre-OST0000-osc-ffff90c142e1b800.idle_timeout=debug osc.lustre-OST0000-osc-ffff90c171daf800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90c142e1b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff90c171daf800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 18:14:55 (1788732895) llite.lustre-ffff90c142e1b800.fast_read=0 llite.lustre-ffff90c171daf800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0168105 s, 125 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff90c142e1b800.fast_read=1 llite.lustre-ffff90c171daf800.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 18:15:10 (1788732910) llite.lustre-ffff90c142e1b800.fast_read=0 llite.lustre-ffff90c171daf800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00246072 s, 5.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff90c142e1b800.fast_read=1 llite.lustre-ffff90c171daf800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 18:15:18 (1788732918) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 18:15:19 (1788732919) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00154203 s, 2.7 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 18:15:22 (1788732922) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0398068 s, 52.7 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 18:15:24 (1788732924) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0358805 s, 58.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 18:15:26 (1788732926) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00178231 s, 6.7 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 18:15:28 (1788732928) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00173181 s, 2.4 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 18:15:31 (1788732931) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00151683 s, 2.7 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 18:15:33 (1788732933) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00155983 s, 2.6 MB/s PASS 101c (3s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 18:15:36 (1788732936) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 b9 11 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0x11b9:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (2s) == sanityn test 103: Test size correctness with lockahead ========================================================== 18:15:38 (1788732938) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788732939 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788732942 Finishing test test23 at 1788732942 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 18:15:46 (1788732946) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 18:15:47 (1788732947) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 18:16:10 (1788732970) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 18:16:11 (1788732971) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00318306 s, 322 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938816 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 18:16:11.000000000 -0400 Modify: 2026-09-06 18:16:11.000000000 -0400 Change: 2026-09-06 18:16:11.000000000 -0400 Birth: 2026-09-06 18:16:11.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938816 -rw-r--r-- 1 0 0 1788732971 1788732971 1788732971 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938816 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 18:16:11.000000000 -0400 Modify: 2026-09-06 18:16:11.000000000 -0400 Change: 2026-09-06 18:16:11.000000000 -0400 Birth: 2026-09-06 18:16:11.000000000 -0400 1788732971 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 18:16:13 (1788732973) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 18:16:15 (1788732975) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.104656 s, 100 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6403 0x1903 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 18:16:20 (1788732980) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0964149 s, 109 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6443 0x192b 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 18:16:28 (1788732988) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.001908 s, 5.2 MB/s llite.lustre-ffff90c142e1b800.hybrid_io=0 llite.lustre-ffff90c171daf800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.20034 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 18:16:35 (1788732995) 192.168.204.150@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 192.168.204.150@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: kmemleak disabled oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: detected 4 online CPUs by sysfs oleg450-server: MODOPTS_LIBCFS= oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 192.168.204.150@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: kmemleak disabled oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: detected 4 online CPUs by sysfs oleg450-server: MODOPTS_LIBCFS= oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 2 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre 192.168.204.150@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg450-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg450-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: kmemleak disabled oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: detected 4 online CPUs by sysfs oleg450-server: MODOPTS_LIBCFS= oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre2 PASS 109 (37s) == sanityn test 110: do not grant another lock on resend ========================================================== 18:17:12 (1788733032) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 18:17:12.000000000 -0400 Modify: 2026-09-06 18:17:12.000000000 -0400 Change: 2026-09-06 18:17:12.000000000 -0400 Birth: 2026-09-06 18:17:12.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg450-client.virtnet oleg450-client.virtnet PASS 110 (168s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 18:20:00 (1788733200) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 18:20:02 (1788733202) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 18:20:03 (1788733203) lustre-OST0000-osc-ffff90c15190d000 active. lustre-OST0001-osc-ffff90c15190d000 active. lustre-OST0000-osc-ffff90c15190d000 active. lustre-OST0001-osc-ffff90c15190d000 active. MGC192.168.204.150@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 18:20:05 (1788733205) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 18:20:06 (1788733206) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 18:20:07 (1788733207) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 18:20:08 (1788733208) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00366466 s, 5.6 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0141845 s, 1.4 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 122: directory size is consistent across mounts ========================================================== 18:20:13 (1788733213) PASS 122 (3s) == sanityn test 200: service remains healthy while able to process request ========================================================== 18:20:16 (1788733216) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=6 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1019 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1039 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1093 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.167 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1709 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1691 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 51.6485 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2656 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2553 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2634 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 84.4485 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg450-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff90c148911800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff90c148911800.ost_server_uuid in IDLE state after 0 sec PASS 200 (91s) cleanup: ====================================================== == sanityn test complete, duration 6994 sec ============== 18:21:48 (1788733308) === sanityn: start cleanup 18:21:48 (1788733308) === Stopping clients: oleg450-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg450-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 18:22:49 (1788733369) ===