-----============= acceptance-small: sanityn ============----- Mon Jun 1 10:03:53 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg439-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 10:04:18 (1780322658) === Starting client oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Mount client oleg439-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Started clients oleg439-client.virtnet: 192.168.204.139@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg439-client.virtnet: executing check_config_client /mnt/lustre oleg439-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg439-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cf706241000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9cf720400800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf706241000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf720400800.idle_timeout=debug disable quota as required oleg439-server: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 10:04:57 (1780322697) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377151 s, 27.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7601] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 10:05:00 (1780322700) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 10:05:08 (1780322708) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 10:05:15 (1780322715) -rw-r--r-- 1 root root 0 Jun 1 10:05 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 10:05:22 (1780322722) -rw-r--r-- 1 root root 0 Jun 1 10:05 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 10:05:30 (1780322730) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 10:05:37 (1780322737) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 10:05:43 (1780322743) total 1 -rwxrwxrwx 1 root root 0 Jun 1 10:05 f2a -rwxrwxrwx 1 root root 0 Jun 1 10:05 f2b total 1 -rwxrwxrwx 1 root root 0 Jun 1 10:05 f2a -rwxrwxrwx 1 root root 0 Jun 1 10:05 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0209819 s, 24.4 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 10:05:50 (1780322750) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 10:05:53 (1780322753) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.381124 s, 5.5 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 10:06:01 (1780322761) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 10:06:08 (1780322768) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 10:06:16 (1780322776) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (9s) == sanityn test 6: remove of open file on other node ============================================================================ 10:06:25 (1780322785) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 10:06:34 (1780322794) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 10:06:42 (1780322802) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 10:06:51 (1780322811) PASS 9a (12s) == sanityn test 9b: append to striped sparse file ======== 10:07:02 (1780322822) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329419 s, 31.8 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0566125 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 10:07:10 (1780322830) 1+0 records in 1+0 records out 1 byte copied, 0.0138488 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0163829 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010005 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0170031 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.051713 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0145384 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0144145 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0161451 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0341195 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102724 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0215973 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0172178 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 10:07:18 (1780322838) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0427713 s, 71.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0546142 s, 75.0 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000306352 s, 10.0 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 10:07:25 (1780322845) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 10:07:33 (1780322853) warning: '-runtime' deprecated, use '-t runtime' instead - link 328 (time 1780322865.45 total 10.02 last 32.73) - link 597 (time 1780322875.53 total 20.11 last 26.67) - link 825 (time 1780322885.58 total 30.16 last 22.69) - link 1086 (time 1780322895.61 total 40.18 last 26.03) - link 1313 (time 1780322905.61 total 50.19 last 22.70) - link 1547 (time 1780322915.61 total 60.19 last 23.40) - link 1784 (time 1780322925.64 total 70.22 last 23.63) - link 2017 (time 1780322935.67 total 80.24 last 23.24) - link 2215 (time 1780322945.71 total 90.28 last 19.72) total: 2449 link in 100.04 seconds: 24.48 ops/second using seed 2091180810 running for 100 seconds - stat 4520 (time 1780322867 ; total 11 ; last 11) - stat 8923 (time 1780322878 ; total 22 ; last 11) - stat 10000 (time 1780322880 ; total 24 ; last 2) - stat 13281 (time 1780322891 ; total 35 ; last 11) - stat 17110 (time 1780322902 ; total 46 ; last 11) - stat 19510 (time 1780322913 ; total 57 ; last 11) - stat 20000 (time 1780322915 ; total 59 ; last 2) - stat 21850 (time 1780322926 ; total 70 ; last 11) - stat 23691 (time 1780322937 ; total 81 ; last 11) - stat 25293 (time 1780322948 ; total 92 ; last 11) total: 26951 stats in 100 seconds: 269.510010 stats/second - unlinked 0 (time 1780322906 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2449) error: No such file or directory total: 2448 unlinks in 98 seconds: 24.979591 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21394) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21396) - No such process PASS 12 (158s) == sanityn test 13: test directory page revocation ======= 10:10:11 (1780323011) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 10:10:21 (1780323021) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 22583 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 10:10:29 (1780323029) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 23163 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (11s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 10:10:39 (1780323039) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23742 Terminated $DIR1/$tdir/sleep 60 PASS 14b (14s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 10:10:54 (1780323054) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24325 Terminated $DIR1/$tdir/sleep 60 PASS 14c (14s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 10:11:08 (1780323068) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24909 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 10:11:18 (1780323078) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (59s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 10:12:17 (1780323137) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.567022 s, 14.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.32805 s, 25.6 MB/s Chance of close/open is 1 in 50 Seed set to 3140 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1780323141.389250 mapwrite 0x2fc418 thru 0x30b87e (0xf467 bytes) truncating to largest ever: 0x39879 000002[0] 1780323141.531562 trunc from 0x30b87f to 0x039878 (0x2d2006 bytes) 000006[0] 1780323141.643178 mapread 0x032a7e thru 0x039878 (0x6dfb bytes) 000007[1] 1780323141.705307 mapread 0x036537 thru 0x039878 (0x3342 bytes) 000008[0] 1780323141.771837 mapread 0x02024b thru 0x029c5b (0x9a11 bytes) 000011[1] 1780323141.860297 write 0x572b3e thru 0x58083a (0xdcfd bytes) 000012[0] 1780323141.894029 mapwrite 0x321628 thru 0x32e4c3 (0xce9c bytes) truncating to largest ever: 0x7e29a7 000014[1] 1780323142.327360 trunc from 0x58083b to 0x7e29a6 (0x26216c bytes) 000016[1] 1780323142.363892 trunc from 0x7e29a7 to 0x781c69 (0x60d3d bytes) truncating to largest ever: 0x914cd5 000019[0] 1780323142.413136 trunc from 0x781c6a to 0x914cd4 (0x19306b bytes) 000024[1] 1780323142.493589 mapwrite 0x4125bd thru 0x418924 (0x6368 bytes) 000027[1] 1780323142.667180 read 0x2fa500 thru 0x30986e (0xf36f bytes) 000028[0] 1780323142.718283 write 0x122135 thru 0x122580 (0x44c bytes) 000029[1] 1780323142.797024 mapread 0x74f420 thru 0x75307a (0x3c5b bytes) 000031[1] 1780323142.871112 read 0x24c4a3 thru 0x25421a (0x7d78 bytes) 000039[0] 1780323142.984947 write 0x9ab581 thru 0x9b08d9 (0x5359 bytes) 000041[0] 1780323143.044839 read 0x205c83 thru 0x20cfae (0x732c bytes) 000042[1] 1780323143.084880 trunc from 0x9b08da to 0x156003 (0x85a8d6 bytes) 000043[1] 1780323143.131882 mapread 0x12a968 thru 0x12cd33 (0x23cc bytes) 000045[0] 1780323143.219216 write 0x9dea2e thru 0x9e7cf7 (0x92ca bytes) 000046[0] 1780323143.261699 mapread 0x8cfa9f thru 0x8d5cd1 (0x6233 bytes) 000047[1] 1780323143.302053 read 0x792c24 thru 0x79efce (0xc3ab bytes) 000049[0] 1780323143.397389 write 0x8ac2c0 thru 0x8b1ef2 (0x5c33 bytes) 000050[0] 1780323143.441061 read 0x0e6cb0 thru 0x0eb1d7 (0x4528 bytes) 000052[1] 1780323143.495749 write 0x9be6cd thru 0x9c6504 (0x7e38 bytes) 000053[0] 1780323143.537986 mapread 0x209f72 thru 0x218f16 (0xefa5 bytes) 000056[1] 1780323143.657320 write 0x0eac34 thru 0x0eef52 (0x431f bytes) 000057[0] 1780323143.725123 write 0x4506bd thru 0x451de1 (0x1725 bytes) 000058[0] 1780323143.754067 trunc from 0x9e7cf8 to 0x883a37 (0x1642c0 bytes) truncating to largest ever: 0x9a0d8f 000060[1] 1780323143.794257 trunc from 0x883a38 to 0x9a0d8e (0x11d357 bytes) 000062[1] 1780323143.881256 trunc from 0x9a0d8f to 0x0411ac (0x95fbe2 bytes) 000063[1] 1780323143.927309 mapread 0x02da1f thru 0x02ed2c (0x130e bytes) 000064[1] 1780323143.941343 mapread 0x03332a thru 0x0397cb (0x64a2 bytes) 000069[1] 1780323144.005660 write 0x4cdc76 thru 0x4d47e4 (0x6b6f bytes) 000070[1] 1780323144.034781 read 0x106c58 thru 0x10aabf (0x3e68 bytes) 000073[1] 1780323144.082331 read 0x3a5639 thru 0x3a59c3 (0x38b bytes) 000075[1] 1780323144.121708 write 0x292e4f thru 0x2a0d6a (0xdf1c bytes) 000079[0] 1780323144.201814 mapwrite 0x86c7d7 thru 0x8789e1 (0xc20b bytes) 000081[1] 1780323144.397720 trunc from 0x8789e2 to 0x0aa3f5 (0x7ce5ec bytes) 000082[1] 1780323144.461139 read 0x031f9b thru 0x03b105 (0x916b bytes) 000083[0] 1780323144.518159 mapread 0x04f58a thru 0x05427b (0x4cf2 bytes) 000092[1] 1780323144.735924 mapread 0x057a09 thru 0x05fa00 (0x7ff8 bytes) 000095[1] 1780323144.818314 write 0x8a6d07 thru 0x8ad9bd (0x6cb7 bytes) 000096[1] 1780323144.843550 mapwrite 0x4ba03b thru 0x4c3810 (0x97d6 bytes) 000097[0] 1780323145.059856 write 0x3593c3 thru 0x35e47d (0x50bb bytes) 000098[1] 1780323145.144193 mapread 0x3c97d2 thru 0x3cbd90 (0x25bf bytes) 000100[1] 1780323145.304090 mapwrite 0x46d094 thru 0x47288e (0x57fb bytes) 000110[0] 1780323145.594321 read 0x35aa15 thru 0x36546b (0xaa57 bytes) 000111[0] 1780323145.602702 mapwrite 0x7869d0 thru 0x7918bd (0xaeee bytes) 000114[1] 1780323145.852155 trunc from 0x8ad9be to 0x48e92a (0x41f093 bytes) 000115[1] 1780323145.893362 mapread 0x19d67e thru 0x1acae2 (0xf465 bytes) 000117[1] 1780323146.091294 read 0x1e3847 thru 0x1e4028 (0x7e2 bytes) 000118[1] 1780323146.150736 trunc from 0x48e92b to 0x6dc8c4 (0x24df9a bytes) 000119[1] 1780323146.217614 mapwrite 0x73e013 thru 0x740f03 (0x2ef1 bytes) 000120[0] 1780323146.319322 mapwrite 0x49bbd3 thru 0x49cbfc (0x102a bytes) 000122[0] 1780323146.476940 trunc from 0x740f04 to 0x43466d (0x30c896 bytes) 000124[0] 1780323146.590650 trunc from 0x43466e to 0x0bb16f (0x3794fe bytes) 000125[0] 1780323146.646873 read 0x042de7 thru 0x045622 (0x283c bytes) 000128[0] 1780323146.714686 write 0x58cdac thru 0x59a344 (0xd599 bytes) 000134[0] 1780323146.815111 trunc from 0x59a345 to 0x80d69e (0x27335a bytes) 000135[1] 1780323146.878238 trunc from 0x80d69f to 0x0b1943 (0x75bd5b bytes) 000137[1] 1780323147.005999 mapread 0x05fbb4 thru 0x0612d5 (0x1722 bytes) 000138[0] 1780323147.061058 read 0x085a69 thru 0x0884c6 (0x2a5e bytes) 000139[1] 1780323147.134854 write 0x75336e thru 0x760e7e (0xdb11 bytes) 000141[1] 1780323147.210473 read 0x3804ba thru 0x38a8bc (0xa403 bytes) 000143[1] 1780323147.271391 mapread 0x17e704 thru 0x18e471 (0xfd6e bytes) 000147[1] 1780323147.380716 write 0x6ae08b thru 0x6b5df7 (0x7d6d bytes) 000148[0] 1780323147.393313 read 0x14dfc7 thru 0x15243e (0x4478 bytes) 000149[1] 1780323147.438795 mapread 0x4be497 thru 0x4cbaf2 (0xd65c bytes) 000150[1] 1780323147.485030 mapread 0x694e7e thru 0x69fd4e (0xaed1 bytes) truncating to largest ever: 0x9b91f9 000153[0] 1780323147.545440 trunc from 0x760e7f to 0x9b91f8 (0x25837a bytes) 000154[1] 1780323147.628267 read 0x299507 thru 0x2a465f (0xb159 bytes) 000156[1] 1780323147.692647 trunc from 0x9b91f9 to 0x1d613d (0x7e30bb bytes) 000157[1] 1780323147.768137 write 0x394c55 thru 0x399acd (0x4e79 bytes) 000159[1] 1780323147.845841 mapread 0x2de704 thru 0x2eb0ff (0xc9fc bytes) 000161[0] 1780323147.906677 trunc from 0x399ace to 0x2b9736 (0xe0397 bytes) 000163[0] 1780323148.025081 mapread 0x230526 thru 0x23ecb1 (0xe78c bytes) 000164[0] 1780323148.093086 write 0x3c337a thru 0x3d2b3d (0xf7c4 bytes) 000166[0] 1780323148.128355 trunc from 0x3d2b3e to 0x3c2bff (0xff3e bytes) 000167[1] 1780323148.159192 write 0x786966 thru 0x7895ab (0x2c46 bytes) 000171[1] 1780323148.257121 write 0x4103c6 thru 0x41fb32 (0xf76d bytes) 000172[0] 1780323148.282502 write 0x5af368 thru 0x5b7307 (0x7fa0 bytes) 000173[1] 1780323148.320552 mapread 0x6a903e thru 0x6b11ad (0x8170 bytes) 000174[1] 1780323148.405632 read 0x2d1d40 thru 0x2d8b96 (0x6e57 bytes) 000176[1] 1780323148.430625 mapwrite 0x4b1203 thru 0x4bbadc (0xa8da bytes) 000182[0] 1780323148.603246 trunc from 0x7895ac to 0x71bed9 (0x6d6d2 bytes) 000185[1] 1780323148.699403 mapwrite 0x62726d thru 0x6335c4 (0xc358 bytes) 000186[1] 1780323148.812661 trunc from 0x71beda to 0x040fa4 (0x6daf35 bytes) 000187[0] 1780323148.849184 mapwrite 0x4a21a2 thru 0x4aa0af (0x7f0e bytes) 000188[1] 1780323149.007973 read 0x3c0d41 thru 0x3c1653 (0x913 bytes) 000189[0] 1780323149.109907 trunc from 0x4aa0b0 to 0x03b564 (0x46eb4b bytes) 000190[0] 1780323149.169243 read 0x00c70a thru 0x01a7f9 (0xe0f0 bytes) 000191[0] 1780323149.208399 mapwrite 0x7354b1 thru 0x743d18 (0xe868 bytes) 000192[1] 1780323149.417552 mapwrite 0x2d5afc thru 0x2dc334 (0x6839 bytes) 000199[1] 1780323149.705517 mapread 0x4c020d thru 0x4ca747 (0xa53b bytes) 000201[0] 1780323149.853675 mapwrite 0x2ba911 thru 0x2c25cb (0x7cbb bytes) 000202[0] 1780323150.040887 mapwrite 0x47392a thru 0x4805f6 (0xcccd bytes) 000204[1] 1780323150.310574 mapread 0x0ae8ec thru 0x0b0cbd (0x23d2 bytes) 000205[0] 1780323150.395115 trunc from 0x743d19 to 0x708868 (0x3b4b0 bytes) 000209[1] 1780323150.536203 mapwrite 0x981d4b thru 0x9845bc (0x2872 bytes) 000212[0] 1780323150.724355 trunc from 0x9845bd to 0x74049b (0x244121 bytes) 000213[1] 1780323150.842921 trunc from 0x74049c to 0x187256 (0x5b9245 bytes) 000218[1] 1780323151.022070 read 0x182d29 thru 0x187256 (0x452e bytes) 000220[1] 1780323151.092065 mapwrite 0x3d0985 thru 0x3dadd4 (0xa450 bytes) 000221[1] 1780323151.296447 write 0x6daf3b thru 0x6e21c8 (0x728e bytes) 000224[0] 1780323151.368970 write 0x3775f7 thru 0x3854ae (0xdeb8 bytes) 000228[0] 1780323151.536501 mapread 0x452d67 thru 0x45bb65 (0x8dff bytes) 000231[1] 1780323151.650501 trunc from 0x6e21c9 to 0x28a6a6 (0x457b22 bytes) 000233[1] 1780323151.703185 read 0x190d66 thru 0x1963d4 (0x566f bytes) 000235[0] 1780323151.782716 mapread 0x1ced12 thru 0x1d424a (0x5539 bytes) 000238[1] 1780323151.885953 mapread 0x09510b thru 0x0a3999 (0xe88f bytes) 000241[0] 1780323151.986427 mapwrite 0x138f6d thru 0x13980f (0x8a3 bytes) 000242[1] 1780323152.096698 mapwrite 0x524a56 thru 0x532699 (0xdc44 bytes) 000243[1] 1780323152.252689 trunc from 0x53269a to 0x83c9cf (0x30a336 bytes) 000245[0] 1780323152.344397 trunc from 0x83c9d0 to 0x66376c (0x1d9263 bytes) 000246[0] 1780323152.409168 read 0x1e0c68 thru 0x1e2066 (0x13ff bytes) 000247[1] 1780323152.475087 trunc from 0x66376d to 0x6460e2 (0x1d68a bytes) 000248[0] 1780323152.562627 mapread 0x56c8c4 thru 0x57043d (0x3b7a bytes) 000249[0] 1780323152.651967 read 0x4bbd83 thru 0x4c5817 (0x9a95 bytes) 000250[1] 1780323152.673562 mapwrite 0x1f6d2c thru 0x20443a (0xd70f bytes) 000251[0] 1780323152.890367 read 0x3973e5 thru 0x3a17d2 (0xa3ee bytes) 000253[1] 1780323152.920546 write 0x25e42d thru 0x266cd8 (0x88ac bytes) 000254[1] 1780323152.996027 write 0x6c9304 thru 0x6cc1cd (0x2eca bytes) 000255[0] 1780323153.039282 mapread 0x5a1671 thru 0x5a2d98 (0x1728 bytes) 000258[0] 1780323153.145960 mapread 0x3ade72 thru 0x3ba16c (0xc2fb bytes) 000259[1] 1780323153.236721 mapwrite 0x221e87 thru 0x223e2a (0x1fa4 bytes) 000262[1] 1780323153.450156 mapread 0x1799d1 thru 0x18057c (0x6bac bytes) 000263[0] 1780323153.537059 read 0x1ee914 thru 0x1f2647 (0x3d34 bytes) 000264[1] 1780323153.572574 read 0x44ab3b thru 0x44b734 (0xbfa bytes) 000266[0] 1780323153.580145 mapread 0x3b7d29 thru 0x3b9590 (0x1868 bytes) 000269[0] 1780323153.608296 mapwrite 0x981e53 thru 0x9879d5 (0x5b83 bytes) 000270[1] 1780323153.750834 mapread 0x1e6564 thru 0x1f3a26 (0xd4c3 bytes) 000271[0] 1780323153.859272 write 0x7568e9 thru 0x757749 (0xe61 bytes) 000275[1] 1780323153.937196 trunc from 0x9879d6 to 0x41ca4d (0x56af88 bytes) 000276[0] 1780323154.019541 mapread 0x045c08 thru 0x04e9f2 (0x8deb bytes) 000277[0] 1780323154.082182 trunc from 0x41ca4e to 0x2fc5c3 (0x12048a bytes) 000283[0] 1780323154.238185 mapwrite 0x167629 thru 0x175417 (0xddef bytes) 000285[1] 1780323154.413884 mapread 0x2ded56 thru 0x2e007c (0x1327 bytes) 000287[0] 1780323154.449966 trunc from 0x2fc5c4 to 0x65c139 (0x35fb76 bytes) 000288[1] 1780323154.506603 read 0x33d86c thru 0x346fb7 (0x974c bytes) 000291[0] 1780323154.551175 trunc from 0x65c13a to 0x0171a3 (0x644f96 bytes) 000292[1] 1780323154.598214 trunc from 0x0171a4 to 0x267e01 (0x250c5e bytes) 000293[1] 1780323154.687118 trunc from 0x267e02 to 0x777303 (0x50f502 bytes) 000294[0] 1780323154.758179 read 0x40a35a thru 0x41479a (0xa441 bytes) 000298[1] 1780323154.846945 mapread 0x715acb thru 0x718578 (0x2aae bytes) 000299[1] 1780323154.917641 trunc from 0x777304 to 0x9b3e5a (0x23cb57 bytes) 000300[0] 1780323155.025657 read 0x39207c thru 0x3944b2 (0x2437 bytes) 000304[1] 1780323155.163245 mapread 0x0fff26 thru 0x10d1a6 (0xd281 bytes) 000305[1] 1780323155.231462 read 0x9312c3 thru 0x93daae (0xc7ec bytes) 000306[1] 1780323155.242175 trunc from 0x9b3e5b to 0x28787d (0x72c5dd bytes) 000309[0] 1780323155.329008 trunc from 0x28787e to 0x67d180 (0x3f5903 bytes) 000310[0] 1780323155.434543 mapwrite 0x64b196 thru 0x65442d (0x9298 bytes) 000311[1] 1780323155.551326 trunc from 0x67d181 to 0x9501a8 (0x2d3028 bytes) 000315[1] 1780323155.638989 mapwrite 0x4dec2d thru 0x4dec71 (0x45 bytes) 000316[0] 1780323155.740050 read 0x10a258 thru 0x10b90a (0x16b3 bytes) 000317[0] 1780323155.799853 mapwrite 0x993a16 thru 0x99b672 (0x7c5d bytes) 000319[1] 1780323156.041518 trunc from 0x99b673 to 0x80db26 (0x18db4c bytes) 000323[0] 1780323156.135524 write 0x5b3427 thru 0x5c269f (0xf279 bytes) 000326[0] 1780323156.206678 write 0x524637 thru 0x5273fb (0x2dc5 bytes) 000327[1] 1780323156.234740 mapread 0x31971f thru 0x31ba7b (0x235d bytes) 000331[0] 1780323156.380955 trunc from 0x80db27 to 0x5aa1e5 (0x263941 bytes) 000334[0] 1780323156.601896 mapwrite 0x87a6a7 thru 0x884451 (0x9dab bytes) 000335[0] 1780323156.765270 mapwrite 0x2c4147 thru 0x2cc320 (0x81da bytes) 000336[1] 1780323156.825271 mapwrite 0x89b587 thru 0x8a72a6 (0xbd20 bytes) 000338[0] 1780323157.047684 trunc from 0x8a72a7 to 0x412e1c (0x49448a bytes) 000339[0] 1780323157.134006 mapread 0x31bace thru 0x32571c (0x9c4f bytes) 000342[1] 1780323157.164326 mapread 0x06c6e6 thru 0x0721a6 (0x5ac1 bytes) 000343[0] 1780323157.243413 write 0x654539 thru 0x6643d4 (0xfe9c bytes) 000350[0] 1780323157.297489 mapread 0x2ab23e thru 0x2b9343 (0xe106 bytes) 000351[0] 1780323157.338273 trunc from 0x6643d5 to 0x495991 (0x1cea43 bytes) 000354[1] 1780323157.381844 write 0x258f2d thru 0x268312 (0xf3e6 bytes) 000356[0] 1780323157.478490 mapwrite 0x88334b thru 0x88c0db (0x8d91 bytes) 000357[1] 1780323157.699514 trunc from 0x88c0dc to 0x2c2c5a (0x5c9481 bytes) 000358[1] 1780323157.733357 mapread 0x1ebf49 thru 0x1ef619 (0x36d1 bytes) 000359[0] 1780323157.766940 mapwrite 0x83eef9 thru 0x843a7f (0x4b87 bytes) 000360[1] 1780323157.903484 trunc from 0x843a80 to 0x2102ab (0x6337d4 bytes) 000361[1] 1780323157.941941 write 0x92f5ca thru 0x936598 (0x6fcf bytes) 000362[0] 1780323157.947887 read 0x3c837e thru 0x3d41ce (0xbe51 bytes) 000363[1] 1780323158.000585 read 0x7a93ba thru 0x7acb9e (0x37e5 bytes) 000364[0] 1780323158.024046 read 0x362c99 thru 0x36b7bd (0x8b25 bytes) 000365[0] 1780323158.032301 trunc from 0x936599 to 0x21ac98 (0x71b900 bytes) 000368[0] 1780323158.126505 write 0x501ab3 thru 0x51117a (0xf6c8 bytes) 000370[1] 1780323158.158662 read 0x3f84b9 thru 0x4048de (0xc426 bytes) 000371[1] 1780323158.292619 trunc from 0x51117b to 0x1e389d (0x32d8dd bytes) 000372[0] 1780323158.443432 write 0x4484f8 thru 0x449ad0 (0x15d9 bytes) 000373[1] 1780323158.511021 mapread 0x206c22 thru 0x207d98 (0x1177 bytes) truncating to largest ever: 0x9dc2a3 000379[0] 1780323158.730272 trunc from 0x449ad1 to 0x9dc2a2 (0x5927d2 bytes) 000380[1] 1780323158.860967 read 0x07f81d thru 0x085d87 (0x656b bytes) 000383[0] 1780323158.919620 read 0x8ef5e8 thru 0x8f8d58 (0x9771 bytes) 000384[0] 1780323158.941500 read 0x27e20b thru 0x287fa9 (0x9d9f bytes) 000385[0] 1780323158.985903 trunc from 0x9dc2a3 to 0x903795 (0xd8b0d bytes) 000386[0] 1780323159.047237 trunc from 0x903796 to 0x548b12 (0x3bac83 bytes) 000388[1] 1780323159.098227 trunc from 0x548b13 to 0x78ee49 (0x246337 bytes) 000389[0] 1780323159.180713 write 0x86923c thru 0x87449b (0xb260 bytes) 000391[0] 1780323159.237403 trunc from 0x87449c to 0x20fa4a (0x664a51 bytes) 000393[0] 1780323159.389968 mapwrite 0x8bc27d thru 0x8c20e5 (0x5e69 bytes) 000394[0] 1780323159.620615 trunc from 0x8c20e6 to 0x902f95 (0x40eb0 bytes) 000395[1] 1780323159.654881 read 0x3b57d7 thru 0x3bbf4f (0x6779 bytes) 000396[0] 1780323159.705463 read 0x8e3e00 thru 0x8ee3c4 (0xa5c5 bytes) 000397[0] 1780323159.725301 mapwrite 0x25bd5c thru 0x25c465 (0x70a bytes) 000399[1] 1780323159.940429 read 0x3918c2 thru 0x394712 (0x2e51 bytes) 000400[1] 1780323159.979685 mapread 0x5a91f1 thru 0x5b7e89 (0xec99 bytes) 000403[1] 1780323160.098979 write 0x0cb9be thru 0x0d0f3b (0x557e bytes) 000404[1] 1780323160.142077 mapread 0x7fcdac thru 0x807b5d (0xadb2 bytes) 000408[0] 1780323160.254614 write 0x96ac6c thru 0x978f76 (0xe30b bytes) 000410[1] 1780323160.291775 write 0x19fe6d thru 0x1ae2c5 (0xe459 bytes) 000416[0] 1780323160.454902 write 0x5bb568 thru 0x5ca696 (0xf12f bytes) 000418[1] 1780323160.538128 mapread 0x557102 thru 0x55b2e1 (0x41e0 bytes) 000419[0] 1780323160.632915 mapwrite 0x110262 thru 0x112b3c (0x28db bytes) 000420[1] 1780323160.810050 mapwrite 0x359e8d thru 0x368784 (0xe8f8 bytes) 000423[0] 1780323160.954646 write 0x293871 thru 0x29649a (0x2c2a bytes) 000424[0] 1780323160.983704 mapwrite 0x299a1a thru 0x29c09d (0x2684 bytes) 000427[0] 1780323161.163664 mapread 0x444066 thru 0x4504ff (0xc49a bytes) 000428[0] 1780323161.244825 write 0x505b4d thru 0x50c7a6 (0x6c5a bytes) 000429[0] 1780323161.323196 trunc from 0x978f77 to 0x4ce89d (0x4aa6d9 bytes) 000432[1] 1780323161.385467 write 0x1c5344 thru 0x1c710f (0x1dcc bytes) 000434[1] 1780323161.491694 mapwrite 0x8063e5 thru 0x807728 (0x1344 bytes) 000436[0] 1780323161.657681 trunc from 0x807729 to 0x92ec1c (0x1274f4 bytes) 000447[1] 1780323162.106073 write 0x92c1f7 thru 0x92f5fb (0x3405 bytes) 000449[1] 1780323162.166888 mapwrite 0x129be8 thru 0x131718 (0x7b31 bytes) 000453[0] 1780323162.373397 read 0x4f9a88 thru 0x4fe501 (0x4a7a bytes) 000454[0] 1780323162.420583 write 0x713d91 thru 0x71419f (0x40f bytes) 000455[1] 1780323162.449500 mapwrite 0x08d7c6 thru 0x09c984 (0xf1bf bytes) 000456[1] 1780323162.701861 mapread 0x46cdd0 thru 0x479249 (0xc47a bytes) 000459[0] 1780323162.807211 mapwrite 0x748db6 thru 0x74febb (0x7106 bytes) 000461[1] 1780323163.027465 trunc from 0x92f5fc to 0x570949 (0x3becb2 bytes) 000463[0] 1780323163.087510 mapwrite 0x4fe8d1 thru 0x504136 (0x5866 bytes) 000465[1] 1780323163.373169 write 0x29835c thru 0x2986ce (0x373 bytes) 000467[1] 1780323163.427391 read 0x17d018 thru 0x17f56f (0x2558 bytes) 000468[1] 1780323163.465178 mapwrite 0x907c3d thru 0x908a0d (0xdd1 bytes) 000470[1] 1780323163.621494 mapwrite 0x7df72a thru 0x7ec7b7 (0xd08e bytes) 000471[0] 1780323163.747302 mapwrite 0x4c7378 thru 0x4d421a (0xcea3 bytes) 000473[1] 1780323163.861202 write 0x2b7fe6 thru 0x2b8977 (0x992 bytes) 000474[1] 1780323163.889324 write 0x36152a thru 0x3697b2 (0x8289 bytes) 000475[1] 1780323163.921872 trunc from 0x908a0e to 0x1d86f0 (0x73031d bytes) 000477[0] 1780323164.018443 trunc from 0x1d86f1 to 0x0e7899 (0xf0e57 bytes) 000481[0] 1780323164.128889 write 0x218f86 thru 0x224f7a (0xbff5 bytes) 000483[1] 1780323164.215736 write 0x4b6b7c thru 0x4b89ff (0x1e84 bytes) 000487[0] 1780323164.392249 write 0x5e9a19 thru 0x5f3b01 (0xa0e9 bytes) 000488[0] 1780323164.489154 mapwrite 0x813d0b thru 0x81c875 (0x8b6b bytes) 000490[0] 1780323164.652347 mapread 0x6a2673 thru 0x6aeef6 (0xc884 bytes) 000492[1] 1780323164.721559 read 0x0ccb0b thru 0x0d4dcf (0x82c5 bytes) 000493[0] 1780323164.742819 read 0x623222 thru 0x626628 (0x3407 bytes) 000494[1] 1780323164.749626 read 0x1c1a20 thru 0x1d1431 (0xfa12 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3164 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1780323165.674090 write 0x408000 thru 0x412989 (0xa98a bytes) 000003[0] 1780323165.691582 mapwrite 0x07b000 thru 0x083651 (0x8652 bytes) 000004[1] 1780323165.838367 read 0x121000 thru 0x127f8e (0x6f8f bytes) 000006[1] 1780323165.861867 write 0x398000 thru 0x3a2507 (0xa508 bytes) truncating to largest ever: 0x6ddca5 000009[1] 1780323165.910996 trunc from 0x41298a to 0x6ddca4 (0x2cb31b bytes) 000014[0] 1780323165.971167 read_OD 0x3e9000 thru 0x3f3fff (0xb000 bytes) 000015[1] 1780323166.031375 mapread 0x65e000 thru 0x668bab (0xabac bytes) 000019[1] 1780323166.074222 mapread 0x5e8000 thru 0x5f785b (0xf85c bytes) 000020[0] 1780323166.117217 read_OD 0x5dc000 thru 0x5e5fff (0xa000 bytes) 000022[0] 1780323166.128860 write_OD 0x73d000 thru 0x749fff (0xd000 bytes) 000024[0] 1780323166.195525 mapwrite 0x021000 thru 0x0233f9 (0x23fa bytes) 000026[0] 1780323166.280816 mapread 0x4a7000 thru 0x4b2e09 (0xbe0a bytes) 000029[1] 1780323166.348208 read 0x049000 thru 0x04b464 (0x2465 bytes) 000031[0] 1780323166.368465 read_OD 0x4bc000 thru 0x4c3fff (0x8000 bytes) 000032[1] 1780323166.376477 write 0x779000 thru 0x77c7d0 (0x37d1 bytes) 000034[1] 1780323166.425205 read 0x73b000 thru 0x7430de (0x80df bytes) 000035[1] 1780323166.465658 mapread 0x4c4000 thru 0x4d17b0 (0xd7b1 bytes) 000038[1] 1780323166.530687 read 0x28f000 thru 0x29a32e (0xb32f bytes) 000039[1] 1780323166.547356 trunc from 0x77c7d1 to 0x0fd0e8 (0x67f6e8 bytes) 000040[1] 1780323166.586978 write 0x91e000 thru 0x91e00b (0xc bytes) 000041[1] 1780323166.598391 trunc from 0x91e00c to 0x611db2 (0x30c259 bytes) 000044[0] 1780323166.639117 mapwrite 0x051000 thru 0x0608f7 (0xf8f8 bytes) 000046[0] 1780323166.762565 trunc from 0x611db3 to 0x63cf84 (0x2b1d2 bytes) 000047[0] 1780323166.803681 mapwrite 0x245000 thru 0x245128 (0x129 bytes) 000049[1] 1780323166.887651 mapread 0x54e000 thru 0x55a1bb (0xc1bc bytes) 000051[0] 1780323166.978599 mapread 0x2c9000 thru 0x2d8989 (0xf98a bytes) 000052[0] 1780323167.001218 trunc from 0x63cf85 to 0x663d4d (0x26dc9 bytes) 000053[1] 1780323167.033549 write 0x11c000 thru 0x11c345 (0x346 bytes) 000055[1] 1780323167.055541 read 0x49a000 thru 0x4a5f57 (0xbf58 bytes) 000056[1] 1780323167.086679 write 0x422000 thru 0x429cb9 (0x7cba bytes) 000060[1] 1780323167.106977 mapread 0x44c000 thru 0x45b211 (0xf212 bytes) 000061[0] 1780323167.140583 write_OD 0x5ad000 thru 0x5adfff (0x1000 bytes) 000062[1] 1780323167.217146 write 0x00b000 thru 0x0134f8 (0x84f9 bytes) 000063[1] 1780323167.237054 read 0x045000 thru 0x04836e (0x336f bytes) 000064[1] 1780323167.275129 read 0x08c000 thru 0x090ecf (0x4ed0 bytes) 000066[1] 1780323167.304702 read 0x58f000 thru 0x59aa53 (0xba54 bytes) 000067[1] 1780323167.319689 read 0x080000 thru 0x0802f0 (0x2f1 bytes) 000068[0] 1780323167.330965 mapread 0x2d5000 thru 0x2e0958 (0xb959 bytes) 000070[0] 1780323167.405932 trunc from 0x663d4e to 0x22a438 (0x439915 bytes) 000072[0] 1780323167.459749 mapread 0x04f000 thru 0x057bb7 (0x8bb8 bytes) 000081[1] 1780323167.629991 read 0x160000 thru 0x16f22a (0xf22b bytes) 000083[0] 1780323167.701752 read_OD 0x0fd000 thru 0x102fff (0x6000 bytes) 000084[1] 1780323167.712106 mapread 0x175000 thru 0x177ab9 (0x2aba bytes) 000086[1] 1780323167.713374 mapread 0x1d1000 thru 0x1de3ad (0xd3ae bytes) 000089[0] 1780323167.716812 mapread 0x1d6000 thru 0x1dd930 (0x7931 bytes) 000091[0] 1780323167.760691 mapwrite 0x82c000 thru 0x8365ba (0xa5bb bytes) 000092[1] 1780323167.933960 mapread 0x70d000 thru 0x70daff (0xb00 bytes) 000095[1] 1780323168.053351 mapwrite 0x5b8000 thru 0x5c2809 (0xa80a bytes) 000097[0] 1780323168.246098 mapread 0x1ec000 thru 0x1edef9 (0x1efa bytes) 000098[1] 1780323168.276972 mapwrite 0x312000 thru 0x31de5e (0xbe5f bytes) 000100[0] 1780323168.399154 mapread 0x245000 thru 0x24f45f (0xa460 bytes) 000101[1] 1780323168.501565 mapwrite 0x339000 thru 0x33a65b (0x165c bytes) truncating to largest ever: 0x8bee13 000102[0] 1780323168.584979 trunc from 0x8365bb to 0x8bee12 (0x88858 bytes) 000105[0] 1780323168.645005 trunc from 0x8bee13 to 0x883947 (0x3b4cb bytes) 000106[0] 1780323168.682508 mapwrite 0x270000 thru 0x274551 (0x4552 bytes) 000107[1] 1780323168.772393 mapwrite 0x0d6000 thru 0x0de7c8 (0x87c9 bytes) 000108[1] 1780323168.889919 write 0x287000 thru 0x28a0a0 (0x30a1 bytes) truncating to largest ever: 0x954f95 000110[0] 1780323168.910430 trunc from 0x883948 to 0x954f94 (0xd164d bytes) 000111[1] 1780323168.981551 mapwrite 0x428000 thru 0x42c9b6 (0x49b7 bytes) 000113[1] 1780323169.137381 write 0x2d6000 thru 0x2d7cb5 (0x1cb6 bytes) 000116[1] 1780323169.182947 read 0x6de000 thru 0x6e6c77 (0x8c78 bytes) 000118[0] 1780323169.221835 write_OD 0x0c7000 thru 0x0d4fff (0xe000 bytes) 000119[1] 1780323169.281382 write 0x013000 thru 0x01482c (0x182d bytes) 000120[0] 1780323169.304053 trunc from 0x954f95 to 0x61a7c0 (0x33a7d4 bytes) 000123[0] 1780323169.352501 trunc from 0x61a7c1 to 0x417942 (0x202e7e bytes) 000124[0] 1780323169.369413 read_OD 0x26a000 thru 0x275fff (0xc000 bytes) 000125[1] 1780323169.393262 mapwrite 0x545000 thru 0x54eb6b (0x9b6c bytes) 000126[1] 1780323169.496902 trunc from 0x54eb6c to 0x09153b (0x4bd630 bytes) 000130[1] 1780323169.561011 mapread 0x079000 thru 0x0791c4 (0x1c5 bytes) 000131[1] 1780323169.582360 read 0x02d000 thru 0x02d28f (0x290 bytes) 000133[0] 1780323169.679407 mapread 0x02e000 thru 0x03cd3b (0xed3c bytes) 000134[1] 1780323169.766675 read 0x014000 thru 0x023564 (0xf565 bytes) 000135[0] 1780323169.812683 write_OD 0x285000 thru 0x287fff (0x3000 bytes) 000138[1] 1780323169.875908 read 0x204000 thru 0x2096b4 (0x56b5 bytes) 000139[0] 1780323169.932150 trunc from 0x288000 to 0x59d66c (0x31566d bytes) 000143[1] 1780323169.997514 mapwrite 0x792000 thru 0x795ea7 (0x3ea8 bytes) 000148[1] 1780323170.255955 write 0x153000 thru 0x155df2 (0x2df3 bytes) 000154[1] 1780323170.323461 write 0x39d000 thru 0x3a15b0 (0x45b1 bytes) 000158[1] 1780323170.361042 mapwrite 0x725000 thru 0x732990 (0xd991 bytes) 000159[1] 1780323170.445924 trunc from 0x795ea8 to 0x46bcc8 (0x32a1df bytes) 000160[1] 1780323170.485247 trunc from 0x46bcc9 to 0x79c1d5 (0x33050d bytes) 000162[1] 1780323170.540280 mapread 0x0ee000 thru 0x0fceab (0xeeac bytes) 000166[0] 1780323170.729814 read_OD 0x3ef000 thru 0x3fbfff (0xd000 bytes) 000168[1] 1780323170.796173 mapread 0x55b000 thru 0x56622e (0xb22f bytes) truncating to largest ever: 0x9e2338 000169[0] 1780323170.912251 trunc from 0x79c1d6 to 0x9e2337 (0x246162 bytes) 000171[1] 1780323170.997948 write 0x518000 thru 0x5201d5 (0x81d6 bytes) 000173[0] 1780323171.058391 write_OD 0x5f8000 thru 0x604fff (0xd000 bytes) 000175[0] 1780323171.127398 read_OD 0x472000 thru 0x47ffff (0xe000 bytes) 000176[1] 1780323171.145074 mapwrite 0x492000 thru 0x49d59b (0xb59c bytes) 000178[0] 1780323171.287145 trunc from 0x9e2338 to 0x49f491 (0x542ea6 bytes) 000179[1] 1780323171.331266 read 0x2bb000 thru 0x2c3f14 (0x8f15 bytes) 000180[1] 1780323171.357597 mapread 0x299000 thru 0x2a8866 (0xf867 bytes) 000182[0] 1780323171.401323 write_OD 0x60e000 thru 0x613fff (0x6000 bytes) 000183[0] 1780323171.502091 mapread 0x0a3000 thru 0x0b034d (0xd34e bytes) 000186[1] 1780323171.561587 trunc from 0x614000 to 0x5fdb40 (0x164bf bytes) 000187[1] 1780323171.613857 trunc from 0x5fdb41 to 0x7e3133 (0x1e55f3 bytes) 000189[1] 1780323171.697350 trunc from 0x7e3134 to 0x5ff379 (0x1e3dba bytes) 000198[1] 1780323171.793357 write 0x71c000 thru 0x726137 (0xa138 bytes) 000200[0] 1780323171.795578 mapwrite 0x728000 thru 0x731428 (0x9429 bytes) 000201[1] 1780323171.925414 mapread 0x5fb000 thru 0x601f86 (0x6f87 bytes) 000202[0] 1780323171.967095 read_OD 0x20a000 thru 0x20efff (0x5000 bytes) 000203[1] 1780323171.982293 write 0x180000 thru 0x18e203 (0xe204 bytes) truncating to largest ever: 0x9e6654 000204[1] 1780323172.000746 trunc from 0x731429 to 0x9e6653 (0x2b522b bytes) 000205[1] 1780323172.049938 read 0x3ee000 thru 0x3f597d (0x797e bytes) 000206[1] 1780323172.117991 read 0x6aa000 thru 0x6ad396 (0x3397 bytes) 000207[1] 1780323172.139172 read 0x91e000 thru 0x928f30 (0xaf31 bytes) 000210[0] 1780323172.224865 mapwrite 0x4b5000 thru 0x4bb109 (0x610a bytes) 000211[1] 1780323172.359734 trunc from 0x9e6654 to 0x68e162 (0x3584f1 bytes) 000215[0] 1780323172.412626 mapread 0x49a000 thru 0x4a3d94 (0x9d95 bytes) 000216[1] 1780323172.456168 write 0x123000 thru 0x125a1f (0x2a20 bytes) 000217[0] 1780323172.487685 mapread 0x5b7000 thru 0x5c4cea (0xdceb bytes) 000218[1] 1780323172.561645 write 0x14d000 thru 0x15031d (0x331e bytes) 000219[0] 1780323172.617078 read_OD 0x1f3000 thru 0x1f7fff (0x5000 bytes) 000220[1] 1780323172.669705 read 0x1f7000 thru 0x1fb3a8 (0x43a9 bytes) 000222[1] 1780323172.696283 mapread 0x291000 thru 0x292ae7 (0x1ae8 bytes) 000227[0] 1780323172.721019 mapwrite 0x88b000 thru 0x89520a (0xa20b bytes) 000229[0] 1780323172.873119 mapwrite 0x095000 thru 0x09f024 (0xa025 bytes) 000231[1] 1780323173.051515 write 0x3db000 thru 0x3e6897 (0xb898 bytes) 000233[0] 1780323173.080845 mapwrite 0x072000 thru 0x073292 (0x1293 bytes) 000235[1] 1780323173.270392 trunc from 0x89520b to 0x014a91 (0x880779 bytes) 000237[0] 1780323173.313363 read_OD 0x009000 thru 0x009fff (0x1000 bytes) 000238[0] 1780323173.366910 mapwrite 0x4db000 thru 0x4de459 (0x345a bytes) 000239[1] 1780323173.559058 mapwrite 0x2ba000 thru 0x2bf2de (0x52df bytes) 000241[1] 1780323173.684002 write 0x3d6000 thru 0x3e1976 (0xb977 bytes) 000242[1] 1780323173.721984 mapread 0x05f000 thru 0x060924 (0x1925 bytes) 000245[1] 1780323173.747439 mapread 0x154000 thru 0x16225c (0xe25d bytes) 000248[1] 1780323173.854295 mapread 0x142000 thru 0x149f1f (0x7f20 bytes) 000249[1] 1780323173.903692 mapread 0x477000 thru 0x477d0e (0xd0f bytes) 000250[0] 1780323173.910675 mapwrite 0x6a5000 thru 0x6a8cb6 (0x3cb7 bytes) 000251[1] 1780323174.074151 mapread 0x15e000 thru 0x16c079 (0xe07a bytes) 000252[1] 1780323174.124178 mapwrite 0x5a8000 thru 0x5aa590 (0x2591 bytes) 000254[1] 1780323174.248282 write 0x5a5000 thru 0x5a9f8b (0x4f8c bytes) 000255[0] 1780323174.265628 trunc from 0x6a8cb7 to 0x23eff0 (0x469cc6 bytes) 000258[0] 1780323174.333694 read_OD 0x1d9000 thru 0x1e7fff (0xf000 bytes) 000259[0] 1780323174.361431 mapread 0x0cb000 thru 0x0cbb67 (0xb68 bytes) 000260[1] 1780323174.392505 trunc from 0x23eff1 to 0x820880 (0x5e1890 bytes) 000261[0] 1780323174.446023 mapread 0x253000 thru 0x25e483 (0xb484 bytes) 000265[1] 1780323174.530363 trunc from 0x820881 to 0x5b3668 (0x26d218 bytes) 000269[0] 1780323174.633719 mapread 0x250000 thru 0x25e7b2 (0xe7b3 bytes) 000271[0] 1780323174.772470 mapwrite 0x600000 thru 0x605516 (0x5517 bytes) 000272[1] 1780323174.951229 write 0x731000 thru 0x738b50 (0x7b51 bytes) 000273[1] 1780323175.010123 read 0x12a000 thru 0x13215c (0x815d bytes) 000274[1] 1780323175.052656 read 0x48f000 thru 0x49c550 (0xd551 bytes) 000275[1] 1780323175.069489 mapread 0x5bf000 thru 0x5bff71 (0xf72 bytes) 000278[1] 1780323175.123857 write 0x00a000 thru 0x00d5b0 (0x35b1 bytes) 000279[1] 1780323175.137213 mapwrite 0x7bd000 thru 0x7bff3d (0x2f3e bytes) 000280[0] 1780323175.271752 write_OD 0x264000 thru 0x26ffff (0xc000 bytes) 000281[0] 1780323175.330897 read_OD 0x249000 thru 0x24efff (0x6000 bytes) 000282[1] 1780323175.350467 write 0x26a000 thru 0x272275 (0x8276 bytes) 000283[0] 1780323175.377711 trunc from 0x7bff3e to 0x5e7cc7 (0x1d8276 bytes) 000286[1] 1780323175.488348 write 0x89c000 thru 0x89c43d (0x43e bytes) 000290[0] 1780323175.560617 write_OD 0x9c9000 thru 0x9d0fff (0x8000 bytes) 000292[1] 1780323175.634610 mapwrite 0x0f4000 thru 0x0f61f5 (0x21f6 bytes) 000293[0] 1780323175.731499 mapread 0x11d000 thru 0x120421 (0x3422 bytes) 000294[1] 1780323175.807109 read 0x56c000 thru 0x5780a2 (0xc0a3 bytes) 000295[1] 1780323175.836713 trunc from 0x9d1000 to 0x74784c (0x2897b3 bytes) 000296[1] 1780323175.905640 read 0x250000 thru 0x25f369 (0xf36a bytes) 000297[1] 1780323175.957684 write 0x62c000 thru 0x632d23 (0x6d24 bytes) 000298[0] 1780323176.016271 trunc from 0x74784d to 0x5d49f5 (0x172e57 bytes) 000299[1] 1780323176.138314 mapread 0x36f000 thru 0x377020 (0x8021 bytes) 000304[1] 1780323176.231740 write 0x841000 thru 0x84f743 (0xe744 bytes) 000307[0] 1780323176.332793 read_OD 0x46c000 thru 0x477fff (0xc000 bytes) 000308[0] 1780323176.392139 read_OD 0x17b000 thru 0x17bfff (0x1000 bytes) 000310[1] 1780323176.406660 mapwrite 0x34a000 thru 0x34ec47 (0x4c48 bytes) 000311[1] 1780323176.635029 read 0x83d000 thru 0x84abf0 (0xdbf1 bytes) 000312[0] 1780323176.646034 read_OD 0x50a000 thru 0x50afff (0x1000 bytes) 000313[0] 1780323176.671544 trunc from 0x84f744 to 0x8fd4ae (0xadd6b bytes) 000316[0] 1780323176.743387 mapread 0x1b8000 thru 0x1b80f9 (0xfa bytes) 000318[0] 1780323176.768552 write_OD 0x1b0000 thru 0x1b3fff (0x4000 bytes) 000320[0] 1780323176.828028 read_OD 0x5a0000 thru 0x5aafff (0xb000 bytes) 000321[1] 1780323176.837513 mapread 0x89c000 thru 0x8a44df (0x84e0 bytes) 000324[1] 1780323176.859514 mapwrite 0x8bf000 thru 0x8c0c8e (0x1c8f bytes) 000328[0] 1780323176.975673 mapwrite 0x68a000 thru 0x6982b3 (0xe2b4 bytes) 000329[0] 1780323177.090251 mapwrite 0x14e000 thru 0x156248 (0x8249 bytes) 000330[1] 1780323177.238636 trunc from 0x8fd4af to 0x1e37d7 (0x719cd7 bytes) 000331[1] 1780323177.356150 mapread 0x021000 thru 0x02f262 (0xe263 bytes) 000334[0] 1780323177.428024 trunc from 0x1e37d8 to 0x5e2c97 (0x3ff4c0 bytes) 000335[1] 1780323177.466271 mapread 0x396000 thru 0x39b347 (0x5348 bytes) 000336[0] 1780323177.533067 mapread 0x233000 thru 0x242e5f (0xfe60 bytes) 000337[1] 1780323177.576578 read 0x42e000 thru 0x432b4c (0x4b4d bytes) 000338[0] 1780323177.596591 read_OD 0x444000 thru 0x448fff (0x5000 bytes) 000340[1] 1780323177.610722 trunc from 0x5e2c98 to 0x3d8072 (0x20ac25 bytes) 000341[1] 1780323177.681540 read 0x2af000 thru 0x2b5ba8 (0x6ba9 bytes) 000342[1] 1780323177.760950 trunc from 0x3d8073 to 0x6a0895 (0x2c8823 bytes) 000344[0] 1780323177.790144 trunc from 0x6a0896 to 0x251d7f (0x44eb16 bytes) 000347[0] 1780323177.870705 write_OD 0x379000 thru 0x37dfff (0x5000 bytes) 000348[0] 1780323177.951812 mapwrite 0x2a2000 thru 0x2ae5d1 (0xc5d2 bytes) 000349[1] 1780323178.089985 trunc from 0x37e000 to 0x56b2f3 (0x1ed2f4 bytes) 000354[0] 1780323178.172649 trunc from 0x56b2f4 to 0x833d87 (0x2c8a94 bytes) 000360[0] 1780323178.300053 write_OD 0x8bb000 thru 0x8bbfff (0x1000 bytes) 000361[1] 1780323178.369021 write 0x4d2000 thru 0x4df512 (0xd513 bytes) 000365[1] 1780323178.466580 mapwrite 0x192000 thru 0x19a547 (0x8548 bytes) 000366[0] 1780323178.557149 read_OD 0x029000 thru 0x036fff (0xe000 bytes) 000367[1] 1780323178.587085 write 0x976000 thru 0x9789c6 (0x29c7 bytes) 000368[1] 1780323178.616670 mapread 0x0a6000 thru 0x0ae990 (0x8991 bytes) 000370[1] 1780323178.725646 mapread 0x79a000 thru 0x7a80d9 (0xe0da bytes) 000375[1] 1780323178.926670 mapwrite 0x589000 thru 0x593642 (0xa643 bytes) 000377[0] 1780323179.039811 mapread 0x3b0000 thru 0x3b6c4e (0x6c4f bytes) 000379[1] 1780323179.091803 write 0x27c000 thru 0x27e35e (0x235f bytes) 000380[1] 1780323179.130355 write 0x57f000 thru 0x589246 (0xa247 bytes) 000381[1] 1780323179.192220 write 0x066000 thru 0x06b99e (0x599f bytes) 000383[1] 1780323179.267811 mapread 0x25a000 thru 0x25ad67 (0xd68 bytes) 000384[0] 1780323179.273855 write_OD 0x7c2000 thru 0x7c2fff (0x1000 bytes) 000385[0] 1780323179.401097 write_OD 0x117000 thru 0x125fff (0xf000 bytes) 000386[1] 1780323179.473946 mapwrite 0x65f000 thru 0x66efa5 (0xffa6 bytes) 000388[0] 1780323179.700748 write_OD 0x80d000 thru 0x818fff (0xc000 bytes) 000390[0] 1780323179.768032 read_OD 0x429000 thru 0x431fff (0x9000 bytes) 000391[0] 1780323179.786362 mapread 0x91c000 thru 0x926d89 (0xad8a bytes) 000393[1] 1780323179.835597 read 0x410000 thru 0x413c73 (0x3c74 bytes) 000396[0] 1780323179.844139 mapread 0x488000 thru 0x48e1b4 (0x61b5 bytes) 000404[1] 1780323179.882925 trunc from 0x9789c7 to 0x6a87d6 (0x2d01f0 bytes) 000407[0] 1780323179.970305 mapread 0x084000 thru 0x08ee2f (0xae30 bytes) 000409[0] 1780323180.035901 trunc from 0x6a87d7 to 0x4d4b46 (0x1d3c90 bytes) 000411[1] 1780323180.107910 trunc from 0x4d4b47 to 0x90e23d (0x4396f7 bytes) 000415[0] 1780323180.224137 write_OD 0x472000 thru 0x478fff (0x7000 bytes) 000417[1] 1780323180.299388 mapread 0x01e000 thru 0x02da09 (0xfa0a bytes) 000418[0] 1780323180.332747 mapwrite 0x782000 thru 0x791250 (0xf251 bytes) 000420[1] 1780323180.480578 mapwrite 0x761000 thru 0x76a2cf (0x92d0 bytes) 000423[0] 1780323180.706705 trunc from 0x90e23e to 0x192706 (0x77bb37 bytes) 000425[0] 1780323180.762667 mapread 0x0fe000 thru 0x0ffb43 (0x1b44 bytes) 000427[0] 1780323180.784238 read_OD 0x06a000 thru 0x06afff (0x1000 bytes) 000431[0] 1780323180.806091 trunc from 0x192707 to 0x48272d (0x2f0027 bytes) 000434[1] 1780323180.873347 read 0x244000 thru 0x253852 (0xf853 bytes) 000441[0] 1780323180.900259 mapwrite 0x408000 thru 0x40b9bb (0x39bc bytes) 000442[1] 1780323181.015059 write 0x7be000 thru 0x7cb429 (0xd42a bytes) 000444[1] 1780323181.036161 read 0x303000 thru 0x30b075 (0x8076 bytes) 000451[0] 1780323181.093419 write_OD 0x9c0000 thru 0x9c8fff (0x9000 bytes) 000452[1] 1780323181.185378 read 0x275000 thru 0x27757e (0x257f bytes) 000454[0] 1780323181.201737 write_OD 0x7b8000 thru 0x7c2fff (0xb000 bytes) 000456[0] 1780323181.288824 mapread 0x8c8000 thru 0x8d4f58 (0xcf59 bytes) 000457[1] 1780323181.320526 write 0x099000 thru 0x099aec (0xaed bytes) 000458[0] 1780323181.357615 mapread 0x4c0000 thru 0x4c6661 (0x6662 bytes) 000459[0] 1780323181.411849 mapread 0x824000 thru 0x82fb7c (0xbb7d bytes) 000463[1] 1780323181.450717 mapwrite 0x6ef000 thru 0x6f33f5 (0x43f6 bytes) 000465[1] 1780323181.623625 mapwrite 0x89a000 thru 0x8a89c2 (0xe9c3 bytes) truncating to largest ever: 0x9ebcf5 000466[0] 1780323181.722697 trunc from 0x9c9000 to 0x9ebcf4 (0x22cf5 bytes) 000468[1] 1780323181.782413 mapread 0x6fd000 thru 0x704f3a (0x7f3b bytes) 000469[1] 1780323181.869950 read 0x142000 thru 0x14596d (0x396e bytes) 000475[0] 1780323181.890033 write_OD 0x586000 thru 0x588fff (0x3000 bytes) 000481[1] 1780323181.985609 mapread 0x2b6000 thru 0x2c35ac (0xd5ad bytes) 000482[0] 1780323182.043524 trunc from 0x9ebcf5 to 0x951126 (0x9abce bytes) 000483[1] 1780323182.110128 write 0x30a000 thru 0x3122d1 (0x82d2 bytes) 000484[0] 1780323182.140240 read_OD 0x6f1000 thru 0x6f7fff (0x7000 bytes) 000487[0] 1780323182.224393 trunc from 0x951127 to 0x6d1c36 (0x27f4f0 bytes) 000488[1] 1780323182.287656 mapwrite 0x31e000 thru 0x3247aa (0x67ab bytes) 000491[1] 1780323182.414668 mapread 0x4c9000 thru 0x4cc71f (0x3720 bytes) 000493[0] 1780323182.463230 write_OD 0x91b000 thru 0x926fff (0xc000 bytes) 000494[0] 1780323182.530716 trunc from 0x927000 to 0x1478d0 (0x7df72f bytes) 000496[0] 1780323182.587832 read_OD 0x076000 thru 0x081fff (0xc000 bytes) All operations completed A-OK! PASS 16a (50s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 10:13:07 (1780323187) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.400817 s, 20.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.315222 s, 26.6 MB/s Chance of close/open is 1 in 50 Seed set to 3189 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x7596af 000004[0] 1780323190.735302 trunc from 00000000 to 0x7596ae (0x7596af bytes) 000005[0] 1780323190.793398 mapread 0x5623ef thru 0x562d06 (0x918 bytes) 000006[0] 1780323190.825126 mapread 0x5c8dd0 thru 0x5ca22f (0x1460 bytes) 000007[1] 1780323190.861209 read 0x3116d6 thru 0x313365 (0x1c90 bytes) 000009[0] 1780323190.914739 trunc from 0x7596af to 0x0d3b35 (0x685b79 bytes) 000010[0] 1780323190.953993 trunc from 0x0d3b36 to 0x163288 (0x8f753 bytes) 000011[1] 1780323191.003456 read 0x051240 thru 0x052221 (0xfe2 bytes) 000012[0] 1780323191.116592 read 0x0e2de1 thru 0x0e3529 (0x749 bytes) 000014[0] 1780323191.138506 mapread 0x0809d0 thru 0x080a54 (0x85 bytes) 000015[1] 1780323191.147559 mapread 0x030c6e thru 0x03238c (0x171f bytes) 000017[1] 1780323191.193640 read 0x098571 thru 0x09987c (0x130c bytes) 000020[0] 1780323191.226823 read 0x0ce5d2 thru 0x0ce844 (0x273 bytes) 000021[1] 1780323191.274597 mapwrite 0x8cbc21 thru 0x8cc016 (0x3f6 bytes) 000022[1] 1780323191.557637 trunc from 0x8cc017 to 0x64c710 (0x27f906 bytes) 000023[0] 1780323191.600380 mapwrite 0x8f2b64 thru 0x8f470c (0x1ba9 bytes) 000024[0] 1780323191.863075 trunc from 0x8f470d to 0x1b5122 (0x73f5ea bytes) 000025[0] 1780323191.907685 mapread 0x049735 thru 0x04b61e (0x1eea bytes) 000027[0] 1780323191.946652 trunc from 0x1b5123 to 0x4f0659 (0x33b537 bytes) 000032[1] 1780323192.050416 read 0x1b2229 thru 0x1b3547 (0x131f bytes) 000035[0] 1780323192.150270 write 0x1e9a8c thru 0x1ead35 (0x12aa bytes) truncating to largest ever: 0x8a9422 000036[0] 1780323192.219319 trunc from 0x4f065a to 0x8a9421 (0x3b8dc8 bytes) 000037[1] 1780323192.316185 mapwrite 0x7760e1 thru 0x7761f4 (0x114 bytes) 000040[0] 1780323192.465745 read 0x071508 thru 0x072779 (0x1272 bytes) 000042[0] 1780323192.529292 mapread 0x24b1c1 thru 0x24c8be (0x16fe bytes) 000045[0] 1780323192.620077 mapread 0x8a7808 thru 0x8a7f1b (0x714 bytes) 000047[0] 1780323192.678111 write 0x1f3115 thru 0x1f4aaa (0x1996 bytes) 000048[0] 1780323192.704741 read 0x352266 thru 0x354082 (0x1e1d bytes) 000049[0] 1780323192.723015 trunc from 0x8a9422 to 0x71aafc (0x18e925 bytes) 000054[1] 1780323192.892746 read 0x3c7f61 thru 0x3c804e (0xee bytes) 000055[1] 1780323192.953462 mapread 0x0cc74a thru 0x0cd537 (0xdee bytes) 000056[0] 1780323193.012710 trunc from 0x71aafd to 0x09ce60 (0x67dc9c bytes) truncating to largest ever: 0x92aad1 000060[0] 1780323193.151281 trunc from 0x09ce61 to 0x92aad0 (0x88dc70 bytes) 000061[1] 1780323193.215277 mapwrite 0x14e90e thru 0x14f4e3 (0xbd6 bytes) 000062[1] 1780323193.343312 mapread 0x5d6520 thru 0x5d7bd0 (0x16b1 bytes) 000064[1] 1780323193.458585 trunc from 0x92aad1 to 0x466c90 (0x4c3e40 bytes) 000066[1] 1780323193.503598 mapread 0x2fae6e thru 0x2fbf34 (0x10c7 bytes) 000067[1] 1780323193.517062 read 0x2eb267 thru 0x2ec7a4 (0x153e bytes) 000068[1] 1780323193.548047 mapread 0x1f45a2 thru 0x1f5b53 (0x15b2 bytes) 000069[0] 1780323193.605140 read 0x275082 thru 0x2760bc (0x103b bytes) 000072[0] 1780323193.666498 write 0x5c70eb thru 0x5c73f9 (0x30f bytes) 000073[1] 1780323193.676490 mapwrite 0x3c4743 thru 0x3c52c8 (0xb86 bytes) 000074[1] 1780323193.799108 mapwrite 0x13a836 thru 0x13b436 (0xc01 bytes) 000075[1] 1780323193.881278 mapwrite 0x68184f thru 0x682f07 (0x16b9 bytes) 000077[0] 1780323194.104825 mapwrite 0x91c14f thru 0x91d51e (0x13d0 bytes) 000078[1] 1780323194.293732 trunc from 0x91d51f to 0x19510f (0x78840f bytes) 000079[1] 1780323194.386700 read 0x160be0 thru 0x160efc (0x31d bytes) 000080[1] 1780323194.463097 trunc from 0x195110 to 0x307275 (0x172166 bytes) 000082[0] 1780323194.496680 write 0x53d82a thru 0x53e0fd (0x8d4 bytes) 000086[0] 1780323194.594319 read 0x318733 thru 0x318dbc (0x68a bytes) 000089[0] 1780323194.646879 write 0x21993e thru 0x21b35d (0x1a20 bytes) 000090[0] 1780323194.712561 mapread 0x242401 thru 0x2433df (0xfdf bytes) 000093[0] 1780323194.751106 mapwrite 0x376a8c thru 0x377487 (0x9fc bytes) 000095[1] 1780323194.901320 read 0x1962d7 thru 0x1964cf (0x1f9 bytes) 000096[0] 1780323194.972971 mapread 0x04b3a2 thru 0x04c1a3 (0xe02 bytes) 000100[0] 1780323195.075744 write 0x20eaed thru 0x20faef (0x1003 bytes) 000105[0] 1780323195.148429 read 0x4cea33 thru 0x4cfdda (0x13a8 bytes) 000107[0] 1780323195.224277 mapread 0x326a15 thru 0x32715e (0x74a bytes) 000109[1] 1780323195.241695 read 0x450553 thru 0x4505d2 (0x80 bytes) 000110[0] 1780323195.303445 mapread 0x20e385 thru 0x20ecfd (0x979 bytes) 000112[1] 1780323195.329139 read 0x4cca8d thru 0x4cd085 (0x5f9 bytes) 000113[0] 1780323195.336751 mapread 0x1497c3 thru 0x14b727 (0x1f65 bytes) 000116[1] 1780323195.376571 write 0x28c0ef thru 0x28d0f2 (0x1004 bytes) 000117[0] 1780323195.417674 write 0x5c5878 thru 0x5c6744 (0xecd bytes) 000120[1] 1780323195.518113 mapread 0x317ada thru 0x318e19 (0x1340 bytes) truncating to largest ever: 0x9d8847 000121[0] 1780323195.578535 trunc from 0x5c6745 to 0x9d8846 (0x412102 bytes) 000127[1] 1780323195.738526 read 0x7b91a1 thru 0x7ba94c (0x17ac bytes) 000128[0] 1780323195.802365 write 0x292b56 thru 0x2939ff (0xeaa bytes) 000129[0] 1780323195.819866 write 0x779532 thru 0x77afcd (0x1a9c bytes) 000130[1] 1780323195.849432 write 0x0cae9a thru 0x0cb68e (0x7f5 bytes) 000134[1] 1780323195.969971 mapread 0x795e7f thru 0x7965a3 (0x725 bytes) 000135[1] 1780323196.022383 write 0x8dc512 thru 0x8dcd2c (0x81b bytes) 000136[1] 1780323196.028464 trunc from 0x9d8847 to 0x538d97 (0x49faaf bytes) 000139[0] 1780323196.087374 mapwrite 0x124546 thru 0x125a9c (0x1557 bytes) 000142[1] 1780323196.200546 mapread 0x1ff668 thru 0x1ffbfe (0x597 bytes) 000143[0] 1780323196.241148 mapread 0x4540e8 thru 0x4546e2 (0x5fb bytes) 000147[0] 1780323196.276406 read 0x404231 thru 0x40498a (0x75a bytes) 000148[1] 1780323196.281700 read 0x378e91 thru 0x37a94e (0x1abe bytes) 000149[1] 1780323196.289607 mapread 0x168b0d thru 0x16a36b (0x185f bytes) 000152[0] 1780323196.331501 mapread 0x2c0cfb thru 0x2c1901 (0xc07 bytes) 000154[1] 1780323196.357537 write 0x1d8474 thru 0x1da02f (0x1bbc bytes) 000156[1] 1780323196.440102 mapread 0x0cadf4 thru 0x0cb825 (0xa32 bytes) 000157[1] 1780323196.457142 trunc from 0x538d98 to 0x8419e2 (0x308c4b bytes) 000158[0] 1780323196.521052 read 0x7cb64e thru 0x7cc3a9 (0xd5c bytes) 000162[0] 1780323196.586226 trunc from 0x8419e3 to 0x18843b (0x6b95a7 bytes) 000163[0] 1780323196.676585 read 0x0cbccf thru 0x0cc935 (0xc67 bytes) 000164[1] 1780323196.778551 trunc from 0x18843c to 0x84736f (0x6bef34 bytes) 000165[0] 1780323196.853939 read 0x00fcb0 thru 0x00ffc9 (0x31a bytes) 000166[1] 1780323196.888952 write 0x844b99 thru 0x846687 (0x1aef bytes) 000168[0] 1780323196.962237 mapwrite 0x3fb7c6 thru 0x3fcf03 (0x173e bytes) 000171[0] 1780323197.123436 mapwrite 0x7bee20 thru 0x7c0174 (0x1355 bytes) 000172[1] 1780323197.231167 read 0x694361 thru 0x694b0d (0x7ad bytes) 000174[0] 1780323197.310320 read 0x845c14 thru 0x846fbc (0x13a9 bytes) 000175[0] 1780323197.342162 write 0x0ed828 thru 0x0ef31f (0x1af8 bytes) 000176[1] 1780323197.364458 mapread 0x1460aa thru 0x14689a (0x7f1 bytes) 000177[1] 1780323197.393685 mapread 0x2eeb14 thru 0x2ef593 (0xa80 bytes) 000178[1] 1780323197.405093 mapread 0x6be66a thru 0x6bed32 (0x6c9 bytes) 000179[0] 1780323197.410498 write 0x78c1ef thru 0x78d378 (0x118a bytes) 000181[0] 1780323197.446197 write 0x868889 thru 0x8692d3 (0xa4b bytes) 000184[0] 1780323197.526856 write 0x1b9c58 thru 0x1bb9ce (0x1d77 bytes) 000185[0] 1780323197.585811 write 0x39be13 thru 0x39bf16 (0x104 bytes) 000188[1] 1780323197.634172 trunc from 0x8692d4 to 0x000ea2 (0x868431 bytes) 000193[0] 1780323197.789065 write 0x857892 thru 0x85953f (0x1cae bytes) 000194[1] 1780323197.823095 mapread 0x2e692f thru 0x2e6aec (0x1be bytes) 000195[1] 1780323197.854505 mapwrite 0x5a5118 thru 0x5a64a1 (0x138a bytes) 000197[0] 1780323198.058876 mapwrite 0x1650ad thru 0x165784 (0x6d8 bytes) 000198[0] 1780323198.174021 write 0x681639 thru 0x681de0 (0x7a8 bytes) 000200[1] 1780323198.243141 mapwrite 0x9fdc9a thru 0x9fe604 (0x96b bytes) 000202[1] 1780323198.391359 mapwrite 0x40c246 thru 0x40cb34 (0x8ef bytes) 000207[1] 1780323198.487002 mapwrite 0x306591 thru 0x307040 (0xab0 bytes) 000208[1] 1780323198.537618 write 0x751cb6 thru 0x752478 (0x7c3 bytes) 000209[1] 1780323198.577685 write 0x9760db thru 0x976c81 (0xba7 bytes) 000210[0] 1780323198.585355 read 0x120632 thru 0x120b28 (0x4f7 bytes) 000211[0] 1780323198.610310 read 0x01e5ad thru 0x01f803 (0x1257 bytes) 000212[0] 1780323198.616110 mapread 0x690bde thru 0x691d1e (0x1141 bytes) 000219[1] 1780323198.660916 write 0x432d6c thru 0x4337a2 (0xa37 bytes) 000221[0] 1780323198.700924 read 0x2e83c6 thru 0x2e8d6e (0x9a9 bytes) 000222[1] 1780323198.707685 trunc from 0x9fe605 to 0x5b54be (0x449146 bytes) 000223[0] 1780323198.793835 mapread 0x539a53 thru 0x53b8d4 (0x1e82 bytes) 000224[1] 1780323198.874211 mapwrite 0x6a11f3 thru 0x6a26fc (0x150a bytes) 000225[0] 1780323198.974852 mapread 0x0c70be thru 0x0c868b (0x15ce bytes) 000227[0] 1780323199.015425 read 0x203663 thru 0x2050ea (0x1a88 bytes) 000228[1] 1780323199.087265 trunc from 0x6a26fd to 0x040416 (0x6622e6 bytes) 000229[0] 1780323199.121732 mapwrite 0x08d5f3 thru 0x08f21d (0x1c2b bytes) 000230[0] 1780323199.285615 mapread 0x073d3b thru 0x074f6f (0x1235 bytes) 000231[1] 1780323199.342727 mapread 0x0521ad thru 0x0527c1 (0x615 bytes) 000232[1] 1780323199.399325 write 0x8a5ce7 thru 0x8a5e48 (0x162 bytes) 000237[0] 1780323199.563601 read 0x11670b thru 0x11758a (0xe80 bytes) 000240[1] 1780323199.645185 trunc from 0x8a5e49 to 0x28ac15 (0x61b233 bytes) 000242[1] 1780323199.745311 read 0x189daf thru 0x18a9b1 (0xc03 bytes) 000243[1] 1780323199.808381 trunc from 0x28ac16 to 0x294335 (0x9720 bytes) 000244[1] 1780323199.858941 write 0x51692d thru 0x516be9 (0x2bd bytes) 000247[0] 1780323199.892462 write 0x864b70 thru 0x86670a (0x1b9b bytes) 000249[1] 1780323199.967607 trunc from 0x86670b to 0x97db16 (0x11740c bytes) 000251[1] 1780323200.079051 mapread 0x1068ba thru 0x108791 (0x1ed8 bytes) 000252[0] 1780323200.142431 trunc from 0x97db17 to 0x1c63cd (0x7b7749 bytes) 000256[1] 1780323200.286672 mapwrite 0x455432 thru 0x456892 (0x1461 bytes) 000258[1] 1780323200.465333 write 0x328548 thru 0x32924a (0xd03 bytes) 000259[1] 1780323200.491940 mapwrite 0x4f4057 thru 0x4f4ae1 (0xa8b bytes) 000260[1] 1780323200.620578 mapread 0x29d9c5 thru 0x29dfdc (0x618 bytes) 000262[1] 1780323200.662878 write 0x0f54e3 thru 0x0f7099 (0x1bb7 bytes) 000263[0] 1780323200.677279 write 0x009f0c thru 0x00b3f1 (0x14e6 bytes) 000266[0] 1780323200.781121 mapread 0x487649 thru 0x488a32 (0x13ea bytes) 000267[0] 1780323200.845466 trunc from 0x4f4ae2 to 0x464a7b (0x90066 bytes) 000268[0] 1780323200.939322 mapread 0x088e46 thru 0x08a4ba (0x1675 bytes) 000271[1] 1780323200.987009 write 0x1192e7 thru 0x11aa6c (0x1786 bytes) 000273[0] 1780323201.039697 mapwrite 0x29d062 thru 0x29d0d8 (0x77 bytes) 000275[0] 1780323201.161965 mapread 0x37eafe thru 0x37fb25 (0x1028 bytes) 000277[0] 1780323201.247945 mapread 0x275134 thru 0x2754cd (0x39a bytes) 000279[1] 1780323201.303302 mapwrite 0x1b0dc3 thru 0x1b1e44 (0x1082 bytes) 000280[0] 1780323201.395584 mapwrite 0x17d267 thru 0x17dfe6 (0xd80 bytes) 000282[1] 1780323201.508346 mapread 0x0e67ee thru 0x0e7c27 (0x143a bytes) 000283[0] 1780323201.577910 mapread 0x1bea9e thru 0x1bf233 (0x796 bytes) 000285[1] 1780323201.628599 mapwrite 0x380136 thru 0x3807d8 (0x6a3 bytes) 000288[1] 1780323201.781125 write 0x22c233 thru 0x22d48c (0x125a bytes) 000289[0] 1780323201.811399 trunc from 0x464a7c to 0x0a5cf7 (0x3bed84 bytes) 000290[1] 1780323201.885029 mapwrite 0x6502d5 thru 0x650c6e (0x99a bytes) 000292[1] 1780323202.098636 write 0x3b4c03 thru 0x3b6a5c (0x1e5a bytes) 000294[1] 1780323202.131262 read 0x631ef4 thru 0x632ac0 (0xbcd bytes) 000301[0] 1780323202.321260 read 0x1619cb thru 0x162775 (0xdab bytes) 000303[0] 1780323202.366670 mapread 0x3ac374 thru 0x3adbeb (0x1878 bytes) 000304[0] 1780323202.391435 write 0x840b48 thru 0x84175f (0xc18 bytes) 000306[0] 1780323202.487214 mapwrite 0x8afd26 thru 0x8afea8 (0x183 bytes) 000307[0] 1780323202.620217 mapread 0x18c83b thru 0x18e3f2 (0x1bb8 bytes) 000308[1] 1780323202.648302 write 0x21d5a5 thru 0x21f50a (0x1f66 bytes) 000312[0] 1780323202.762562 mapread 0x2dfc80 thru 0x2e15f8 (0x1979 bytes) 000313[0] 1780323202.830415 write 0x8645b1 thru 0x865328 (0xd78 bytes) 000314[1] 1780323202.856395 write 0x04b090 thru 0x04b456 (0x3c7 bytes) 000315[1] 1780323202.888450 read 0x841cf4 thru 0x843386 (0x1693 bytes) 000316[1] 1780323202.910628 write 0x6e857a thru 0x6e94db (0xf62 bytes) 000319[0] 1780323202.979971 mapwrite 0x6fb141 thru 0x6fc0ca (0xf8a bytes) 000324[0] 1780323203.319129 trunc from 0x8afea9 to 0x132426 (0x77da82 bytes) truncating to largest ever: 0x9f8503 000327[0] 1780323203.434088 trunc from 0x132427 to 0x9f8502 (0x8c60dc bytes) 000329[1] 1780323203.499940 trunc from 0x9f8503 to 0x383dad (0x674755 bytes) 000331[1] 1780323203.594327 mapwrite 0x91e98f thru 0x92093b (0x1fad bytes) 000332[0] 1780323203.755683 trunc from 0x92093c to 0x0300d8 (0x8f0863 bytes) 000334[1] 1780323203.830057 write 0x7bd667 thru 0x7be300 (0xc9a bytes) 000335[0] 1780323203.856563 mapread 0x21917e thru 0x219d4a (0xbcd bytes) 000337[1] 1780323203.922626 mapread 0x594286 thru 0x59624b (0x1fc6 bytes) 000338[1] 1780323203.978184 mapread 0x1056fc thru 0x1069a3 (0x12a8 bytes) 000340[1] 1780323204.023541 write 0x45fa9e thru 0x460135 (0x698 bytes) 000344[0] 1780323204.081420 trunc from 0x7be301 to 0x0f8aba (0x6c5846 bytes) 000346[1] 1780323204.137733 mapread 0x015908 thru 0x017292 (0x198b bytes) 000347[0] 1780323204.182249 mapwrite 0x16888a thru 0x169187 (0x8fe bytes) 000348[0] 1780323204.326861 mapread 0x162ff1 thru 0x164458 (0x1468 bytes) 000349[1] 1780323204.350432 mapread 0x0b2ff6 thru 0x0b4272 (0x127d bytes) 000353[0] 1780323204.458087 trunc from 0x169188 to 0x5ed101 (0x483f7a bytes) 000355[0] 1780323204.515616 write 0x83f827 thru 0x840973 (0x114d bytes) 000358[0] 1780323204.659831 mapwrite 0x01de32 thru 0x01fd9c (0x1f6b bytes) 000360[0] 1780323204.820600 write 0x939fae thru 0x93ad40 (0xd93 bytes) 000361[1] 1780323204.849841 mapwrite 0x3823ba thru 0x383200 (0xe47 bytes) 000362[0] 1780323204.952196 write 0x256c69 thru 0x256f16 (0x2ae bytes) 000363[1] 1780323204.967157 write 0x4edfa5 thru 0x4ee478 (0x4d4 bytes) 000364[1] 1780323205.004211 write 0x11d048 thru 0x11eac4 (0x1a7d bytes) 000368[0] 1780323205.061641 mapwrite 0x5afdfa thru 0x5affea (0x1f1 bytes) 000370[1] 1780323205.196376 trunc from 0x93ad41 to 0x4428cf (0x4f8471 bytes) 000371[1] 1780323205.290496 mapread 0x1729fd thru 0x1742e8 (0x18ec bytes) 000372[0] 1780323205.332845 trunc from 0x4428d0 to 0x1899ba (0x2b8f15 bytes) 000374[1] 1780323205.420947 read 0x06cfbb thru 0x06ef18 (0x1f5e bytes) 000376[0] 1780323205.520362 mapwrite 0x8ca20d thru 0x8cb37c (0x1170 bytes) 000380[0] 1780323205.694923 mapread 0x7c218f thru 0x7c26be (0x530 bytes) 000381[1] 1780323205.699327 mapwrite 0x8b94d8 thru 0x8b9934 (0x45d bytes) 000382[0] 1780323205.826619 write 0x11f796 thru 0x1215d5 (0x1e40 bytes) 000383[0] 1780323205.853805 mapread 0x3e94b3 thru 0x3ea796 (0x12e4 bytes) 000384[1] 1780323205.876209 trunc from 0x8cb37d to 0x5abb57 (0x31f825 bytes) 000385[1] 1780323205.928049 mapwrite 0x00f47b thru 0x011138 (0x1cbe bytes) 000388[0] 1780323206.033749 read 0x0ecbf1 thru 0x0ecf30 (0x340 bytes) 000389[0] 1780323206.050219 write 0x062788 thru 0x062898 (0x111 bytes) 000391[0] 1780323206.088907 trunc from 0x5abb58 to 0x90abce (0x35f077 bytes) 000392[1] 1780323206.186830 trunc from 0x90abcf to 0x9c012e (0xb5560 bytes) 000394[0] 1780323206.279179 trunc from 0x9c012f to 0x63d24a (0x382ee4 bytes) 000396[1] 1780323206.317491 write 0x6a1910 thru 0x6a29c1 (0x10b2 bytes) 000405[1] 1780323206.502602 read 0x46bfe5 thru 0x46df7b (0x1f97 bytes) 000409[0] 1780323206.613058 trunc from 0x6a29c2 to 0x406c78 (0x29bd49 bytes) 000410[0] 1780323206.662003 trunc from 0x406c79 to 0x41f783 (0x18b0b bytes) 000411[1] 1780323206.692696 trunc from 0x41f784 to 0x8fcc46 (0x4dd4c3 bytes) 000412[1] 1780323206.782568 mapread 0x12149d thru 0x122a85 (0x15e9 bytes) 000417[1] 1780323206.946554 mapread 0x277958 thru 0x278c39 (0x12e2 bytes) 000418[1] 1780323206.957387 read 0x1277fe thru 0x127c75 (0x478 bytes) 000419[1] 1780323206.963074 read 0x061a88 thru 0x0638fe (0x1e77 bytes) 000420[1] 1780323206.987722 mapwrite 0x24f770 thru 0x2502cc (0xb5d bytes) 000421[0] 1780323207.045045 mapwrite 0x22ec8f thru 0x22f5d5 (0x947 bytes) 000422[0] 1780323207.174213 mapwrite 0x4e9374 thru 0x4e9bfc (0x889 bytes) 000424[0] 1780323207.292518 mapwrite 0x6a2d96 thru 0x6a45da (0x1845 bytes) 000427[0] 1780323207.411605 mapwrite 0x2461b1 thru 0x247e0b (0x1c5b bytes) 000428[0] 1780323207.511564 read 0x5d2b76 thru 0x5d450c (0x1997 bytes) 000432[0] 1780323207.537248 mapwrite 0x2d23f2 thru 0x2d3c41 (0x1850 bytes) 000438[0] 1780323207.702126 mapwrite 0x9546b8 thru 0x95620d (0x1b56 bytes) 000442[0] 1780323207.834962 trunc from 0x95620e to 0x235eba (0x720353 bytes) 000445[0] 1780323207.906513 mapread 0x009d19 thru 0x00aac2 (0xdaa bytes) 000447[1] 1780323207.919234 trunc from 0x235ebb to 0x0e6534 (0x14f986 bytes) 000448[1] 1780323207.965734 trunc from 0x0e6535 to 0x0ad0a4 (0x39490 bytes) 000450[0] 1780323208.054673 mapwrite 0x4046d7 thru 0x40669e (0x1fc8 bytes) 000451[0] 1780323208.227583 mapwrite 0x28edc9 thru 0x28fef1 (0x1129 bytes) 000452[1] 1780323208.297596 mapwrite 0x15e972 thru 0x1607db (0x1e6a bytes) 000453[1] 1780323208.383367 trunc from 0x40669f to 0x210adf (0x1f5bbf bytes) 000454[0] 1780323208.461942 read 0x02e3f3 thru 0x02fac1 (0x16cf bytes) 000457[0] 1780323208.531872 mapwrite 0x239bac thru 0x23ae94 (0x12e9 bytes) 000459[0] 1780323208.702098 mapread 0x160636 thru 0x1607d4 (0x19f bytes) 000463[1] 1780323208.772561 trunc from 0x23ae95 to 0x1593f1 (0xe1aa3 bytes) 000466[1] 1780323208.922044 write 0x5ee65c thru 0x5ef87f (0x1224 bytes) 000469[0] 1780323208.926516 mapwrite 0x56db8b thru 0x56e403 (0x879 bytes) 000470[1] 1780323209.060039 mapread 0x05aad9 thru 0x05c624 (0x1b4c bytes) 000471[1] 1780323209.134136 trunc from 0x5ef880 to 0x1713b4 (0x47e4cb bytes) 000472[0] 1780323209.188515 read 0x12491f thru 0x1249a9 (0x8b bytes) 000477[1] 1780323209.279184 read 0x010100 thru 0x011040 (0xf41 bytes) 000481[0] 1780323209.315902 read 0x161571 thru 0x161f54 (0x9e4 bytes) 000484[1] 1780323209.342499 read 0x0ac0ba thru 0x0ac172 (0xb9 bytes) 000486[1] 1780323209.379166 mapread 0x08400e thru 0x085ae0 (0x1ad3 bytes) 000487[0] 1780323209.380747 write 0x6fe2e1 thru 0x6ff170 (0xe90 bytes) 000488[1] 1780323209.405620 mapwrite 0x33296b thru 0x334400 (0x1a96 bytes) 000489[0] 1780323209.617914 trunc from 0x6ff171 to 0x839ac0 (0x13a950 bytes) 000490[1] 1780323209.678049 write 0x792642 thru 0x7927db (0x19a bytes) 000491[0] 1780323209.718901 write 0x6e7a14 thru 0x6e7fdf (0x5cc bytes) 000494[1] 1780323209.854050 read 0x11182d thru 0x1118de (0xb2 bytes) 000497[0] 1780323209.956396 write 0x581ad6 thru 0x581db7 (0x2e2 bytes) 000499[1] 1780323210.093589 mapwrite 0x4b82de thru 0x4b9dd4 (0x1af7 bytes) 000500[1] 1780323210.267534 write 0x57973f thru 0x57aacd (0x138f bytes) All operations completed A-OK! PASS 16b (27s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 10:13:34 (1780323214) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (3s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 10:13:37 (1780323217) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.355301 s, 23.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.274445 s, 30.6 MB/s ldlm.namespaces.MGC192.168.204.139@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9cf706241000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9cf720400800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9cf720400800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9cf720400800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0779 s, 10.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.98343 s, 35.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.33561 s, 44.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.43305 s, 19.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.45706 s, 12.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.03028 s, 34.6 MB/s PASS 16d (44s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 10:14:21 (1780323261) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.22317 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130651 s, 8.0 MB/s PASS 16e (5s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 10:14:27 (1780323267) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 10:14:53 (1780323293) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 10:15:19 (1780323319) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (5s) == sanityn test 16i: read after truncate file ============ 10:15:24 (1780323324) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (5s) == sanityn test 16j: race dio with buffered i/o ========== 10:15:29 (1780323329) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.60192 s, 16.4 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.774213 s, 33.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.56117 s, 16.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.06892 s, 24.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.42573 s, 18.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.33539 s, 19.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.70353 s, 15.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.19065 s, 22.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.21077 s, 21.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.69809 s, 15.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.83206 s, 14.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.33612 s, 19.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.42264 s, 18.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.69424 s, 15.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.77074 s, 14.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.70314 s, 15.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.90789 s, 13.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.94668 s, 13.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.0076 s, 13.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (23s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 10:15:52 (1780323352) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Jun 1 10:16:15 EDT 2026 All operations completed A-OK! + fsx 2 done: rc=0 Mon Jun 1 10:16:15 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Mon Jun 1 10:16:15 EDT 2026 + fsx 4 done: rc=0 Mon Jun 1 10:16:15 EDT 2026 + fsx 5 done: rc=0 Mon Jun 1 10:16:15 EDT 2026 PASS 16k (26s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 10:16:18 (1780323378) fail_loc=0x8000030a PASS 17 (6s) == sanityn test 18: mmap sanity check =========================================================================================== 10:16:24 (1780323384) mmap test1: basic mmap operation (PASS, 0.046562s) mmap test2: MAP_PRIVATE not write back (PASS, 0.040905s) mmap test3: concurrent mmap ops on two nodes (PASS, 8.299s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.27s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.11105s) mmap test6: check mmap write/read content on two nodes (PASS, 0.081434s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.58805s) mmap test11: multi-thread shared mmap (PASS, 0.5328s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0316s) PASS 18 (19s) == sanityn test 19: test concurrent uncached read races ========================================================================= 10:16:43 (1780323403) oleg439-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 10:16:45 (1780323405) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 10:16:48 (1780323408) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 10:16:50 (1780323410) now is 1780323411 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 new atime is 1780323472 PASS 23 (64s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 10:17:55 (1780323475) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2209792 2176 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2209792 2176 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 389 524.5K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 389 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 389 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 389 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537471 389 537082 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234917 389 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2209792 2176 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 389 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 389 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537471 389 537082 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234917 389 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537471 389 537082 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234917 389 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2209792 2176 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2209792 2176 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (4s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 10:17:58 (1780323478) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 10:18:00 (1780323480) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (3s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 10:18:03 (1780323483) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 10:18:04 (1780323484) utime: good mknod times 1780323484 <= 1780323485 <= 1780323485 for /mnt/lustre/f26a utime: good mknod times 1780323484 <= 1780323485 <= 1780323485 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 10:18:07 (1780323487) PASS 26b (5s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 10:18:12 (1780323492) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502396 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-01 10:18:13.000000000 -0400 Birth: 2026-06-01 10:18:12.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502396 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-01 10:18:13.000000000 -0400 Birth: 2026-06-01 10:18:12.000000000 -0400 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 10:18:16 (1780323496) 0 PASS 30 (5s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 10:18:21 (1780323501) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 10:18:24 (1780323504) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg439-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.139@tcp:/lustre 7542784 38912 7499776 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 10:18:37 (1780323517) -rw-r--r-- 1 root root 0 Jun 1 10:18 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 mv: failed to access '/mnt/lustre2/f31r.sanityn.target': Input/output error PASS 31r (5s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 10:18:42 (1780323522) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 43068 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 10:18:45 (1780323525) PASS 31t (3s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 10:18:48 (1780323528) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 10:18:50 (1780323530) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 10:18:51 (1780323531) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 10:18:52 (1780323532) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 10:18:53 (1780323533) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cf706241000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf706241000.ost_server_uuid in FULL state after 0 sec oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf706241000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cf706241000.ost_server_uuid in IDLE state after 0 sec oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cf706241000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf706241000.ost_server_uuid in FULL state after 0 sec oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf706241000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cf706241000.ost_server_uuid in IDLE state after 0 sec oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cf706241000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf706241000.ost_server_uuid in FULL state after 0 sec oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9cf706241000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9cf706241000.ost_server_uuid in IDLE state after 0 sec PASS 34 (92s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 10:20:25 (1780323625) fail_loc=0x80000317 Race attempt 0 Wait for 49114 49245 for 60 sec... total: 4000 open/close in 9.62 seconds: 415.65 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.139@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.204.139@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 PASS 35 (64s) == sanityn test 36: handle ESTALE/open-unlink correctly == 10:21:30 (1780323690) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg439-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.39022 s, 134 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7447552) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.391222 s, 134 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.37736 s, 139 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7498752) after_dd(7447552) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.450165 s, 116 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7498752) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.891296 s, 58.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.82205 s, 28.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33061 s, 39.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.61137 s, 32.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.866536 s, 60.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7496704) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.871192 s, 60.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7491584) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.86856 s, 60.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7443456) after(7494656) PASS 36 (280s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 10:26:09 (1780323969) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 - create 1820 (time 1780323982.02 total 10.00 last 181.97) - create 3367 (time 1780323992.03 total 20.01 last 154.61) - create 4996 (time 1780324002.04 total 30.02 last 162.76) - create 6419 (time 1780324012.04 total 40.02 last 142.29) - create 7942 (time 1780324022.04 total 50.02 last 152.24) - create 9588 (time 1780324032.04 total 60.02 last 164.54) total: 10000 create in 62.96 seconds: 158.84 ops/second PASS 37 (71s) == sanityn test 39a: file mtime does not change after rename ========================================================== 10:27:20 (1780324040) repeat after cancel_lru_locks PASS 39a (5s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 10:27:26 (1780324046) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 10:27:32 (1780324052) repeat after cancel_lru_locks PASS 39c (6s) == sanityn test 39d: sync write should update mtime ====== 10:27:38 (1780324058) fail_loc=0x411 fail_loc=0 PASS 39d (5s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 10:27:44 (1780324064) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (13s) == sanityn test 40c: pdirops: link and others ======================================================================== 10:27:57 (1780324077) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (11s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 10:28:08 (1780324088) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (13s) == sanityn test 40e: pdirops: rename and others ======================================================================== 10:28:21 (1780324101) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (12s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 10:28:33 (1780324113) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (9s) == sanityn test 41b: pdirops: create vs create ======================================================================== 10:28:42 (1780324122) open: File exists Conflict PASS 41b (8s) == sanityn test 41c: pdirops: create vs link ======================================================================== 10:28:50 (1780324130) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (8s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 10:28:58 (1780324138) Conflict PASS 41d (8s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 10:29:07 (1780324147) Conflict PASS 41e (9s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 10:29:16 (1780324156) Conflict PASS 41f (8s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 10:29:24 (1780324164) Conflict PASS 41g (9s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 10:29:33 (1780324173) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (8s) == sanityn test 41i: reint_open: create vs create ======== 10:29:41 (1780324181) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (822s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 10:43:23 (1780325003) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 10:43:29 (1780325009) open: File exists Conflict PASS 42b (7s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 10:43:36 (1780325016) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 10:43:42 (1780325022) Conflict PASS 42d (7s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 10:43:49 (1780325029) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (7s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 10:43:56 (1780325036) Conflict PASS 42f (7s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 10:44:03 (1780325043) Conflict PASS 42g (7s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 10:44:10 (1780325050) Conflict PASS 42h (7s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 10:44:17 (1780325057) PASS 43a (35s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 10:44:52 (1780325092) Conflict PASS 43b (7s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 10:44:59 (1780325099) Conflict PASS 43c (7s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 10:45:06 (1780325106) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (6s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 10:45:12 (1780325112) Conflict PASS 43e (7s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 10:45:19 (1780325119) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (9s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 10:45:28 (1780325128) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (8s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 10:45:36 (1780325136) Conflict PASS 43h (7s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 10:45:43 (1780325143) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 10:45:45 (1780325145) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (58s) == sanityn test 43k: unlink vs create ==================== 10:46:43 (1780325203) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (584s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 10:56:27 (1780325787) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Jun 1 10:56:32 EDT 2026 PASS 44a (7s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 10:56:34 (1780325794) open: File exists Conflict PASS 44b (7s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 10:56:41 (1780325801) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (7s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 10:56:48 (1780325808) Conflict PASS 44d (7s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 10:56:55 (1780325815) Conflict PASS 44e (8s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 10:57:03 (1780325823) Conflict PASS 44f (7s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 10:57:10 (1780325830) Conflict PASS 44g (7s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 10:57:17 (1780325837) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 10:57:23 (1780325843) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 10:57:25 (1780325845) PASS 45a (51s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 10:58:17 (1780325897) Conflict PASS 45b (7s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 10:58:23 (1780325903) Conflict PASS 45c (7s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 10:58:30 (1780325910) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (6s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 10:58:36 (1780325916) Conflict PASS 45e (7s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 10:58:43 (1780325923) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (7s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 10:58:50 (1780325930) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (6s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 10:58:56 (1780325936) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 10:59:01 (1780325941) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 10:59:03 (1780325943) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (449s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 11:06:32 (1780326392) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 11:06:38 (1780326398) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 11:06:43 (1780326403) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 11:06:48 (1780326408) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 11:06:54 (1780326414) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 11:06:59 (1780326419) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 11:07:04 (1780326424) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 11:07:09 (1780326429) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 11:07:14 (1780326434) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 11:07:16 (1780326436) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 11:07:16 (1780326436) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 11:07:17 (1780326437) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 11:07:19 (1780326439) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 11:07:20 (1780326440) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (0s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 11:07:21 (1780326441) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (0s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 11:07:22 (1780326442) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (0s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 11:07:23 (1780326443) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00604135 s, 1.7 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 11:07:30 (1780326450) 0+1 records in 0+1 records out 158 bytes copied, 0.00764982 s, 20.7 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (217562) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 11:07:34 (1780326454) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00486435 s, 211 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018449 s, 56.8 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 11:07:49 (1780326469) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00452921 s, 226 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 11:07:55 (1780326475) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178511 s, 58.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 11:07:58 (1780326478) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 11:08:03 (1780326483) llite.lustre-ffff9cf706241000.xattr_cache=1 llite.lustre-ffff9cf720400800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505009 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:08:03.000000000 -0400 Modify: 2026-06-01 11:08:03.000000000 -0400 Change: 2026-06-01 11:08:03.000000000 -0400 Birth: 2026-06-01 11:08:03.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505012 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:08:08.000000000 -0400 Modify: 2026-06-01 11:08:09.000000000 -0400 Change: 2026-06-01 11:08:09.000000000 -0400 Birth: 2026-06-01 11:08:08.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:08:14.000000000 -0400 Modify: 2026-06-01 11:08:14.000000000 -0400 Change: 2026-06-01 11:08:14.000000000 -0400 Birth: 2026-06-01 11:08:14.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:08:14.000000000 -0400 Modify: 2026-06-01 11:08:14.000000000 -0400 Change: 2026-06-01 11:08:14.000000000 -0400 Birth: 2026-06-01 11:08:14.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:08:19.000000000 -0400 Modify: 2026-06-01 11:08:20.000000000 -0400 Change: 2026-06-01 11:08:20.000000000 -0400 Birth: 2026-06-01 11:08:19.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:08:19.000000000 -0400 Modify: 2026-06-01 11:08:20.000000000 -0400 Change: 2026-06-01 11:08:20.000000000 -0400 Birth: 2026-06-01 11:08:19.000000000 -0400 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 11:08:27 (1780326507) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (7s) == sanityn test 55b: rename vs unlink source dir ========= 11:08:34 (1780326514) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 11:08:42 (1780326522) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (12s) == sanityn test 55d: rename file vs link ================= 11:08:54 (1780326534) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 11:09:04 (1780326544) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 11:09:05 (1780326545) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 11:09:18 (1780326558) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 11:09:32 (1780326572) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1780326573 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.7745 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (25s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 11:09:57 (1780326597) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1780326598 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.39095 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (40s) == sanityn test 60: Verify data_version behaviour ======== 11:10:37 (1780326637) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 11:10:40 (1780326640) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 11:10:42 (1780326642) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 11:10:45 (1780326645) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0038295 s, 17.1 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 11:10:47 (1780326647) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00573487 s, 11.4 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 11:10:48 (1780326648) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 11:10:49 (1780326649) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0049408 s, 13.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 11:10:50 (1780326650) llite.lustre-ffff9cf706241000.xattr_cache=1 llite.lustre-ffff9cf720400800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 11:10:52 (1780326652) llite.lustre-ffff9cf706241000.xattr_cache=1 llite.lustre-ffff9cf720400800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 11:10:54 (1780326654) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00247027 s, 415 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0035398 s, 289 kB/s 230328: get lock2 230331: get lock1 230328: done 230328: get lock1 230331: done 230331: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 230331: cannot get lock2: Resource deadlock avoided 230331: exit rc=0 fcntl cmd 7 failed: Resource deadlock avoided 230328: cannot get lock1: Resource deadlock avoided 230328: done 230328: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 11:11:01 (1780326661) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0214115 s, 97.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127007 s, 82.6 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 230910 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 11:11:08 (1780326668) opening files: ........ closing files: ........ PASS 76 (19s) == sanityn test 77a: check FIFO NRS policy =============== 11:11:27 (1780326687) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.134195 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463009 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587997 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615465 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061951 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802879 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10037 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994509 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100501 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988593 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962616 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116793 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122924 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119812 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115622 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12858 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102638 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178577 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124968 s, 83.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211905 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193482 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156269 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245968 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147585 s, 71.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101094 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256611 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158311 s, 66.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181435 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00472204 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390823 s, 268 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178467 s, 58.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231166 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164189 s, 63.9 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 11:11:31 (1780326691) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.141945 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07038 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689695 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875649 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087994 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859932 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904854 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717044 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807719 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959285 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808289 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108838 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961896 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100224 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105094 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111979 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106523 s, 9.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234114 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016904 s, 62.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0046523 s, 225 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189092 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449359 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045676 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486639 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257203 s, 40.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246679 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270395 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03795 s, 27.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355725 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00302315 s, 347 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178792 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155799 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114893 s, 91.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.119796 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022421 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482665 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444514 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646743 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691964 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622661 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499604 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628746 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632619 s, 16.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938083 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105869 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121607 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119859 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118948 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131061 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111158 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00539847 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158713 s, 66.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118217 s, 88.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146378 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171607 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224634 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292913 s, 35.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230451 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352204 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298277 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213998 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216298 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251742 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303428 s, 34.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02752 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324387 s, 32.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 11:11:35 (1780326695) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151835 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281883 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658355 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644738 s, 16.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767451 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810672 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731883 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784595 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974032 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109677 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127948 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127659 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12446 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129497 s, 8.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114729 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132751 s, 79.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17224 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225891 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201568 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022759 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227462 s, 46.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886619 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388648 s, 27.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189472 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0048041 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354742 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163337 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345552 s, 30.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038224 s, 274 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344919 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100097 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142872 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00293431 s, 357 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.13889 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036842 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641081 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063209 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948959 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880784 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845521 s, 12.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652068 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800941 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701459 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113055 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104626 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110842 s, 9.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111199 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109573 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101897 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124544 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201287 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137132 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212205 s, 49.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126673 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716419 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200956 s, 52.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227253 s, 46.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220673 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023347 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541523 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0073081 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136837 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00948688 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00986183 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00733404 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164186 s, 63.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.120898 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025479 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576662 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895573 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114059 s, 9.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120133 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132255 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14898 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154935 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163085 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154836 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15865 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16416 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153076 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162938 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180731 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158111 s, 6.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165677 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209109 s, 50.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308348 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254331 s, 41.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032006 s, 32.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278564 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174877 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329387 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339541 s, 30.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147215 s, 71.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207157 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241033 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198359 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337712 s, 31.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414788 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399644 s, 26.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 11:11:40 (1780326700) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.128949 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217373 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655917 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768389 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717478 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850442 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703626 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081477 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873645 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863077 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733966 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105934 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105519 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786436 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844072 s, 12.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936576 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900744 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148387 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196779 s, 53.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147348 s, 71.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163545 s, 64.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105847 s, 99.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170313 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210299 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202332 s, 51.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00468532 s, 224 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241521 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117659 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00856923 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028646 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360366 s, 291 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00834431 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00916839 s, 114 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.126527 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314698 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438299 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336098 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636675 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399225 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829333 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105766 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1035 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963038 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117265 s, 8.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976018 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108418 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103323 s, 10.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656783 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109177 s, 9.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119716 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142638 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187469 s, 55.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105516 s, 99.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00868876 s, 121 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021429 s, 48.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257658 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116315 s, 90.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00815321 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267894 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00475056 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033773 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0091094 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122559 s, 85.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460768 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145879 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140528 s, 74.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.124138 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245172 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537646 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625932 s, 16.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599133 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583445 s, 18.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819852 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436044 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546484 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832135 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858995 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805833 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821096 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750438 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827605 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975513 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112267 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024112 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273485 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157681 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257107 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217608 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170125 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200202 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160258 s, 65.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013936 s, 75.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169378 s, 61.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220127 s, 47.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153507 s, 68.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215971 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178635 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00987809 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149667 s, 70.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 11:11:45 (1780326705) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.39@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.133767 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306523 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251393 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639287 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819174 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129239 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14622 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117516 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129947 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115592 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127186 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141298 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124201 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120923 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11974 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125401 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122701 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206608 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230547 s, 45.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237617 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181634 s, 57.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261253 s, 40.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256918 s, 40.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308882 s, 33.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129577 s, 80.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00642658 s, 163 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382404 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130336 s, 80.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246255 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382759 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184468 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161922 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124125 s, 84.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.124421 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328066 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679278 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638874 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670528 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613844 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675982 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761276 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678189 s, 15.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863306 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993156 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776147 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102158 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964448 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939829 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888745 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990576 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153777 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164826 s, 63.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238013 s, 44.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187546 s, 55.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018689 s, 56.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293454 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172388 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256563 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026623 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00346394 s, 303 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234332 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241332 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00717501 s, 146 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731726 s, 143 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409669 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145571 s, 72.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147838 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028465 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323124 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100611 s, 10.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106412 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110735 s, 9.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115628 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111647 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108108 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128824 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151574 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133818 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127554 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158527 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147457 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146447 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160707 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188505 s, 55.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173626 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153143 s, 68.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263621 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289673 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250061 s, 41.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279836 s, 37.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145377 s, 72.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179999 s, 58.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252328 s, 41.6 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00913615 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608199 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731729 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019871 s, 52.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339881 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145369 s, 72.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.144011 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341442 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703496 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057631 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957336 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103054 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927657 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929411 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108437 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934344 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997575 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985581 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950155 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904635 s, 11.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915802 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923221 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100708 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122129 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140627 s, 74.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188422 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019037 s, 55.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252156 s, 41.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114553 s, 91.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102867 s, 102 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157755 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262767 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361865 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218003 s, 48.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171261 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032755 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189258 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00650627 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152694 s, 68.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 11:11:53 (1780326713) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145152 s, 116 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170291 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035356 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274501 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486687 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536797 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594299 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559665 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642484 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785599 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726067 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085497 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843985 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098544 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813711 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845028 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105939 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01545 s, 67.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120383 s, 87.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101938 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162757 s, 64.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170919 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142532 s, 73.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212057 s, 49.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175162 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163069 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188148 s, 55.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00383976 s, 273 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02159 s, 48.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207725 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478926 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00625037 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00168102 s, 624 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14531 s, 115 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383577 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349563 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381768 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710304 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786718 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834739 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732412 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749258 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784008 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740673 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907363 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105512 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880301 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757639 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939357 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11369 s, 9.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038432 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185705 s, 56.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153408 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137427 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218334 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207413 s, 50.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020014 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249096 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236901 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219923 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387859 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221923 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0078344 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240423 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356916 s, 294 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500185 s, 210 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14063 s, 119 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293843 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518347 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427043 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554452 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278941 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818217 s, 12.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793743 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102856 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106655 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969997 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914063 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950641 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106319 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107219 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104158 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106025 s, 9.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198167 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185528 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195421 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197889 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185653 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130703 s, 80.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181653 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00611899 s, 171 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013373 s, 78.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136191 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204708 s, 51.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215945 s, 48.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266406 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392555 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340178 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00489267 s, 214 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130997 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024695 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041733 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727249 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106963 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112478 s, 9.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103999 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801358 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921492 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103517 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117446 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10799 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112719 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108547 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116727 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138129 s, 7.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133968 s, 7.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf706241000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665034 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0066208 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266359 s, 39.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162475 s, 64.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253981 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036089 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035342 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117106 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403818 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172128 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420671 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340332 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112537 s, 93.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394644 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032727 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481623 s, 21.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (7s) == sanityn test 77g: Change TBF type directly ============ 11:12:00 (1780326720) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 11:12:03 (1780326723) oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg439-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg439-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg439-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg439-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 11:12:07 (1780326727) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 11:12:14 (1780326734) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.107932 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037303 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562071 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574674 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059944 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734134 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609013 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566103 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542249 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770358 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640238 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692871 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770386 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768941 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056756 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651774 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719741 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113082 s, 92.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147373 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126714 s, 82.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017221 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155837 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144034 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00307316 s, 341 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395003 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185857 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193558 s, 54.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181597 s, 57.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195613 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350104 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255575 s, 410 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218077 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00231204 s, 454 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93358 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4525 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (53s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 11:13:08 (1780326788) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26161 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105845 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742586 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692126 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112757 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145546 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20369 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159658 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21464 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202462 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213813 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208255 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188395 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176449 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206784 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228294 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250573 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475679 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047428 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495576 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322875 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466178 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497494 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558166 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655962 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597744 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561357 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669711 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627361 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651802 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663286 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691435 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807081 s, 13.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7759 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2336 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3227 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127305 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131844 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208476 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157878 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115666 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207316 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210191 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235326 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175211 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259928 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245903 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216223 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292747 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247909 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308379 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311962 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03421 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665256 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634399 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633068 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798844 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795785 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527463 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837265 s, 12.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771806 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715879 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883685 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582204 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084536 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552266 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966314 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959331 s, 10.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8887 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9612 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (144s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 11:15:32 (1780326932) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24528 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281227 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067843 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869399 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117006 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128652 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116809 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113513 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13026 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148105 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160856 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187673 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18198 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160601 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190973 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198007 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214126 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239348 s, 43.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303791 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340753 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266508 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368238 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246916 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272096 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290306 s, 36.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267932 s, 39.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459575 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358252 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299767 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342274 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277632 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339602 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433696 s, 24.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8534 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9453 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26098 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112955 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128861 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132788 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124924 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102832 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157647 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164637 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759681 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136182 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155127 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139611 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132067 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185156 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121629 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19912 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156107 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409866 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577894 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539327 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250653 s, 41.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385304 s, 27.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319543 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457688 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660003 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574315 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471591 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568794 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749434 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639139 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493587 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529192 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579594 s, 18.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8271 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9183 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (147s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 11:17:59 (1780327079) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.496333 s, 33.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358304 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442912 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571723 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160091 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164764 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148136 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144916 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155445 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161186 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166207 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133234 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165509 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179551 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173639 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133844 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18819 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126453 s, 82.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193279 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260429 s, 40.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420912 s, 249 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422161 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427503 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00331035 s, 317 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364719 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421708 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395175 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00433192 s, 242 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043795 s, 239 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00748337 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028394 s, 36.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298473 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313889 s, 33.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.34499 s, 16.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82522 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.39@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.39@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.500829 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602304 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042247 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601255 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455189 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539359 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111805 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101043 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133298 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137163 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123568 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15508 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162322 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185325 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19544 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201139 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241959 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351732 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369152 s, 28.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208961 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239568 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528986 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457092 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432698 s, 24.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396089 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040129 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744124 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531034 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442451 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438739 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472905 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061661 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485612 s, 21.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.67426 s, 18.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86224 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.39@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.599409 s, 28.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686145 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467993 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701955 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617907 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464433 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138013 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126434 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143895 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410401 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185449 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162141 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235336 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170489 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268361 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273738 s, 3.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9cf706241000.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323521 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239663 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038938 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860428 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899943 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133179 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767601 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613309 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518172 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00668303 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490419 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426528 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366483 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632785 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333244 s, 31.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198522 s, 52.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262909 s, 39.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.05036 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13953 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.55742 s, 30.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417023 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05206 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999693 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119518 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147434 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156015 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162296 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170998 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161483 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156577 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174653 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163869 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184554 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175217 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302918 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295531 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149142 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302408 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444205 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438612 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374415 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401484 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059362 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790201 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560093 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351907 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486524 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062057 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507298 s, 20.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937443 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518566 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585087 s, 17.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1393 s, 10.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90073 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.61998 s, 18.7 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82489 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.927499 s, 18.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502862 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123912 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122884 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841441 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927503 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123749 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126414 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173393 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151344 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143816 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165734 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179082 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172948 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185409 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21564 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231586 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270646 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00620238 s, 169 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170339 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748577 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697115 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641326 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681925 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555261 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862299 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00676562 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610885 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346757 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306718 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248649 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198414 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166475 s, 63.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6879 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5321 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.320245 s, 52.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412092 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515355 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568096 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457409 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511359 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463483 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270667 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683836 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626973 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115416 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10897 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693466 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130336 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096436 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10481 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10247 s, 10.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438334 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393061 s, 26.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036312 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156131 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373797 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349013 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214102 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220775 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308084 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333854 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245425 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269552 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374107 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282176 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289927 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325157 s, 32.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.14326 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79533 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.537755 s, 31.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292516 s, 35.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470949 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474677 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118089 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113519 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134493 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141007 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148191 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11712 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121892 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140571 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142094 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141652 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1843 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197724 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188208 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498413 s, 210 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.005592 s, 188 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236626 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132274 s, 79.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232739 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00551661 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249008 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266043 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440662 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224196 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305957 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0055272 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506293 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544737 s, 192 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00595001 s, 176 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626443 s, 167 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92804 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81144 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87562 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80516 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (402s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 11:24:41 (1780327481) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.392234 s, 42.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555192 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054646 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557515 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443868 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801072 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614347 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869191 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112371 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115814 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10786 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12118 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120252 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126213 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156041 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15799 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174769 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387985 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231919 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493193 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136806 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381819 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193173 s, 54.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415299 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238642 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256117 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300399 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295902 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714636 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038508 s, 27.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389868 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423734 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465338 s, 22.5 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.56686 s, 18.8 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80197 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (40s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 11:25:21 (1780327521) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.39@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.39@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.272318 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609874 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065194 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477984 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913274 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125152 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155283 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172679 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168241 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171892 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152642 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180172 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190232 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164038 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169127 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201787 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156046 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012915 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194303 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00325455 s, 322 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490885 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407572 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051308 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411838 s, 255 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708082 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00566742 s, 185 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551822 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205962 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023521 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155323 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171755 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174128 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132996 s, 78.8 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.78425 s, 18.1 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85313 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (42s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 11:26:03 (1780327563) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.39@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.287579 s, 58.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433253 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639213 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720428 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329804 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553531 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571556 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593968 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807703 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037568 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146264 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134046 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101071 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160711 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116345 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162749 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175041 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286171 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478486 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379365 s, 276 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447543 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00265738 s, 395 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00361636 s, 290 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00333832 s, 314 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634082 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670126 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327667 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038808 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00372443 s, 282 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00795506 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448564 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334003 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00352888 s, 297 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.58946 s, 18.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97079 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (36s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 11:26:40 (1780327600) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.525305 s, 31.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427092 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518539 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982383 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843101 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106716 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12711 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968029 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990113 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618049 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102789 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101596 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10552 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154016 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121742 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209391 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237079 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248222 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046538 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350346 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391617 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233178 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560176 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437583 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445114 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332624 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596533 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364186 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381381 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358158 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303249 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374153 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455076 s, 23.0 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91498 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83464 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03591 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82244 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (84s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 11:28:04 (1780327684) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.904767 s, 18.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526729 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619882 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048077 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667913 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512506 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817581 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631105 s, 16.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673313 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128868 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117405 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151683 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143803 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203248 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150427 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154746 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19034 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382641 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294819 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261725 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273341 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286963 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347759 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386587 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325584 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394593 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396694 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361927 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414936 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561191 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418969 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425198 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505708 s, 20.7 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6142 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9448 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 11:29:11 (1780327751) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.355604 s, 47.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360877 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383424 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482466 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727296 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797998 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109651 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117917 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118295 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110696 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129887 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109686 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144439 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118209 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144808 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141071 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133652 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374099 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203379 s, 51.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00707204 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165636 s, 63.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138712 s, 75.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458282 s, 229 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186405 s, 56.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387802 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047206 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374769 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025052 s, 41.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252962 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00462982 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00414146 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379508 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00280101 s, 374 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13265 s, 20.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78383 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg439-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25001 s, 67.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291475 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027848 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854469 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095156 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103152 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937091 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936202 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110828 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903514 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107575 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119059 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997582 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189507 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188163 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244026 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247242 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378974 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122503 s, 85.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0091538 s, 115 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00869693 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460627 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00415069 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259499 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427821 s, 245 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396013 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00376456 s, 279 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0065074 s, 161 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00673541 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156404 s, 67.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00601918 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00310013 s, 338 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111347 s, 94.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93676 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82351 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5864 s, 8.3 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5513 s, 9.9 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (138s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 11:31:29 (1780327889) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:) Stopping client oleg439-client.virtnet /mnt/lustre opts: Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg439-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg439-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.39,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg439-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.39,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg439-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.39,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/64540/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.39,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg439-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg439-server: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg439-server: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg439-server: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Started clients oleg439-client.virtnet: 192.168.204.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Started clients oleg439-client.virtnet: 192.168.204.139@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cf720130800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9cf721840800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf720130800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf721840800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.477802 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639202 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852635 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106639 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927799 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138629 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963547 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091798 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157337 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149246 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493213 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165592 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128417 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117741 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763989 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105604 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153162 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307312 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761095 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755624 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754923 s, 13.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056032 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574625 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612374 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722704 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893267 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667343 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108166 s, 96.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766332 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114029 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102591 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915126 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918551 s, 11.4 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.98659 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.2326 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.678462 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725407 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137188 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137114 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611705 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160937 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186068 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07539 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117082 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541752 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256755 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278409 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168607 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178752 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284096 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291507 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323618 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908847 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464507 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00721256 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00533474 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740437 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757168 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764003 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784396 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533409 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00280431 s, 374 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00777356 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024385 s, 430 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637765 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682582 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100936 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00529799 s, 198 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg439-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg439-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.48136 s, 14.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95162 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (218s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 11:35:08 (1780328108) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg439-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 11:35:26 (1780328126) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 11:35:36 (1780328136) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg439-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.42358 s, 306 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg439-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.4319 s, 259 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (60s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 11:36:36 (1780328196) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.394611 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759155 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928272 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104787 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163057 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210636 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235054 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189095 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232058 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232963 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204293 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309097 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322459 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341135 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242812 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331908 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf720130800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9cf720130800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316458 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596654 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629314 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697909 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865335 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736582 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965957 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774874 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078489 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105078 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843799 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863307 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881769 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116115 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990021 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116339 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110985 s, 9.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.5192 s, 7.8 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.64545 s, 18.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.812093 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196986 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101549 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12845 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349897 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328234 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397772 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352174 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391677 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35132 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342069 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403141 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426142 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429138 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419133 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406406 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43278 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9cf720130800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9cf720130800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173022 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179301 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170098 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170255 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17903 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203931 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165591 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159677 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191986 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172462 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207292 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205289 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178847 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158244 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203637 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192493 s, 5.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg439-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg439-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.379 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.52208 s, 19.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (103s) == sanityn test 77o: Changing rank should not panic ====== 11:38:19 (1780328299) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 11:38:33 (1780328313) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (94s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 11:40:07 (1780328407) mds.MDS.mdt.nrs_policies=tbf oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg439-client: oleg439-server: ssh exited with exit code 36 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 oleg439-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg439-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg439-client: oleg439-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (27s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 11:40:34 (1780328434) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 270948 kill md_thread xargs PASS 77r (56s) == sanityn test 77s: Check TBF LRU shrinker ============== 11:41:30 (1780328490) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (68s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 11:42:38 (1780328558) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 11:42:48 (1780328568) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935726 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:42:49.000000000 -0400 Modify: 2026-06-01 11:42:49.000000000 -0400 Change: 2026-06-01 11:42:49.000000000 -0400 Birth: 2026-06-01 11:42:49.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 11:43:06 (1780328586) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 11:43:09 (1780328589) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 11:43:13 (1780328593) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 11:43:16 (1780328596) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 11:43:20 (1780328600) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 11:43:23 (1780328603) - open/close 749 (time 1780328618.21 total 10.01 last 74.86) - open/close 1458 (time 1780328628.22 total 20.02 last 70.83) - open/close 2268 (time 1780328638.22 total 30.02 last 80.98) total: 2500 open/close in 32.70 seconds: 76.46 ops/second snapshot_time 1780328728.852692171 secs.nsecs start_time 1780328605.676832162 secs.nsecs elapsed_time 123.175860009 secs.nsecs open 2505 samples [usecs] 162 19776 4060862 10656721000 close 2504 samples [usecs] 147 11709 1008315 991361265 mknod 2501 samples [usecs] 521 19666 3587565 8623917905 unlink 2500 samples [usecs] 784 115122 6370733 92812483859 mkdir 2 samples [usecs] 1100 3286 4386 12007796 rmdir 2 samples [usecs] 14069 14719 28788 414585722 rename 2500 samples [usecs] 503 60134 18777589 302991688585 getattr 3760 samples [usecs] 7 16786 236486 540952960 parallel_rename_file 2500 samples [usecs] 503 60134 18777589 302991688585 crossdir_rename 2500 samples [usecs] 503 60134 18777589 302991688585 PASS 81d (129s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 11:45:32 (1780328732) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7601 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7601 PASS 82 (5s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 11:45:37 (1780328737) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 11:45:41 (1780328741) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 11:45:54 (1780328754) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1780328756 Finishing test test1 at 1780328763 PASS 85 (15s) == sanityn test 90: open/create and unlink striped directory ========================================================== 11:46:09 (1780328769) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 11:46:14 (1780328774) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (5s) == sanityn test 92: create remote directory under orphan directory ========================================================== 11:46:18 (1780328778) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 11:46:22 (1780328782) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5858 0x16e2 0x240000400 1 5822 0x16be 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5859 0x16e3 0x240000400 1 5823 0x16bf 0x280000400 rc1=0 and rc2=0 PASS 93 (15s) == sanityn test 94: signal vs CP callback race =========== 11:46:37 (1780328797) osc.lustre-OST0000-osc-ffff9cf720130800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9cf721840800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9cf720130800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9cf721840800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.169879 s, 48.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.01127 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6178: 288209 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0860655 s, 47.6 kB/s osc.lustre-OST0000-osc-ffff9cf720130800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9cf721840800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf720130800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf721840800.idle_timeout=debug PASS 94 (14s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 11:46:51 (1780328811) llite.lustre-ffff9cf720130800.fast_read=0 llite.lustre-ffff9cf721840800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0934708 s, 22.4 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9cf720130800.fast_read=1 llite.lustre-ffff9cf721840800.fast_read=1 PASS 95a (19s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 11:47:11 (1780328831) llite.lustre-ffff9cf720130800.fast_read=0 llite.lustre-ffff9cf721840800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00426915 s, 2.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9cf720130800.fast_read=1 llite.lustre-ffff9cf721840800.fast_read=1 PASS 95b (12s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 11:47:22 (1780328842) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 11:47:25 (1780328845) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00618397 s, 662 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (6s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 11:47:30 (1780328850) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134393 s, 15.6 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (5s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 11:47:35 (1780328855) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130416 s, 16.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (5s) == sanityn test 100e: DoM: read on open and file size ==== 11:47:40 (1780328860) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0067241 s, 1.8 MB/s PASS 100e (5s) == sanityn test 101a: Discard DoM data on unlink ========= 11:47:45 (1780328865) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00377873 s, 1.1 MB/s PASS 101a (5s) == sanityn test 101b: Discard DoM data on rename ========= 11:47:50 (1780328870) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00676107 s, 606 kB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 11:47:56 (1780328876) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0109522 s, 374 kB/s PASS 101c (6s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 11:48:04 (1780328884) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1780328885 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1780328889 Finishing test test23 at 1780328889 PASS 103 (11s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 11:48:14 (1780328894) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 11:48:17 (1780328897) fail_loc=0x416 PASS 105 (20s) == sanityn test 106a: Verify the btime via statx() ======= 11:48:37 (1780328917) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 11:48:40 (1780328920) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.027166 s, 37.7 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938261 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:48:41.000000000 -0400 Modify: 2026-06-01 11:48:41.000000000 -0400 Change: 2026-06-01 11:48:41.000000000 -0400 Birth: 2026-06-01 11:48:40.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938261 -rw-r--r-- 1 0 0 1780328920 1780328921 1780328921 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938261 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:48:41.000000000 -0400 Modify: 2026-06-01 11:48:41.000000000 -0400 Change: 2026-06-01 11:48:41.000000000 -0400 Birth: 2026-06-01 11:48:40.000000000 -0400 1780328921 1024 1 PASS 106b (4s) == sanityn test 106c: Verify statx attributes mask ======= 11:48:45 (1780328925) PASS 106c (5s) == sanityn test 107a: Basic grouplock conflict =========== 11:48:49 (1780328929) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.141356 s, 74.2 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5829 0x16c5 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 First grouplock blocks second one PASS 107a (6s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 11:48:55 (1780328935) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.14878 s, 70.5 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5868 0x16ec 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7601 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (10s) == sanityn test 108a: lseek: parallel updates ============ 11:49:05 (1780328945) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00669222 s, 1.5 MB/s llite.lustre-ffff9cf720130800.hybrid_io=0 llite.lustre-ffff9cf721840800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.45094 s, 1.9 MB/s PASS 108a (9s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 11:49:14 (1780328954) 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre (opts:) 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 192.168.204.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.139@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg439-server: detected 4 online CPUs by sysfs oleg439-server: MODOPTS_LIBCFS= oleg439-server: Force libcfs to create 2 CPU partitions Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre2 PASS 109 (49s) == sanityn test 110: do not grant another lock on resend ========================================================== 11:50:03 (1780329003) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-01 11:50:04.000000000 -0400 Modify: 2026-06-01 11:50:04.000000000 -0400 Change: 2026-06-01 11:50:04.000000000 -0400 Birth: 2026-06-01 11:50:04.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg439-client.virtnet oleg439-client.virtnet PASS 110 (170s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 11:52:53 (1780329173) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 11:52:54 (1780329174) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 11:52:55 (1780329175) lustre-OST0000-osc-ffff9cf720136800 active. lustre-OST0001-osc-ffff9cf720136800 active. lustre-OST0000-osc-ffff9cf720136800 active. lustre-OST0001-osc-ffff9cf720136800 active. MGC192.168.204.139@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 11:52:58 (1780329178) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 11:53:00 (1780329180) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 11:53:01 (1780329181) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 117a: TCU: Init and enable Trash Can on MDTs ========================================================== 11:53:03 (1780329183) SKIP: sanityn test_117a ldiskfs only test SKIP 117a (1s) == sanityn test 117b: Move regular file and empty dir into trash can dir ========================================================== 11:53:04 (1780329184) SKIP: sanityn test_117b ldiskfs only test SKIP 117b (1s) == sanityn test 117c: Move deleted tree with multiple levels into trash ========================================================== 11:53:05 (1780329185) SKIP: sanityn test_117c ldiskfs only test SKIP 117c (2s) == sanityn test 117d: Per-User Trash can Type testing ==== 11:53:07 (1780329187) SKIP: sanityn test_117d ldiskfs only test SKIP 117d (1s) == sanityn test 117e: Undeleted dir in trash should keep its original xattrs ========================================================== 11:53:08 (1780329188) SKIP: sanityn test_117e ldiskfs only test SKIP 117e (2s) == sanityn test 117f: Uncache the dentry under the trash dir ========================================================== 11:53:10 (1780329190) SKIP: sanityn test_117f ldiskfs only test SKIP 117f (1s) == sanityn test 117g: Access .Trash for a non-striped directory ========================================================== 11:53:11 (1780329191) SKIP: sanityn test_117g ldiskfs only test SKIP 117g (2s) == sanityn test 121: trunc append race =================== 11:53:13 (1780329193) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00508782 s, 4.0 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0248704 s, 823 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (5s) == sanityn test 200: service remains healthy while able to process request ========================================================== 11:53:18 (1780329198) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1018 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1167 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1235 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1149 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1205 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1335 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1273 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1266 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1626 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1463 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1702 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.18 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1675 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1951 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2203 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2626 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2332 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2639 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9cf707f14800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9cf707f14800.ost_server_uuid in IDLE state after 0 sec PASS 200 (68s) cleanup: ====================================================== == sanityn test complete, duration 6631 sec ============== 11:54:26 (1780329266) === sanityn: start cleanup 11:54:27 (1780329267) === Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg439-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 11:55:45 (1780329345) ===