-----============= acceptance-small: sanityn ============----- Wed Sep 16 21:14:40 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg216-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 21:15:06 (1789607706) === Starting client oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 Mount client oleg216-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 Started clients oleg216-client.virtnet: 192.168.202.116@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg216-client.virtnet: executing check_config_client /mnt/lustre oleg216-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg216-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fe893100000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9fe8a00ea000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fe893100000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fe8a00ea000.idle_timeout=debug disable quota as required oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 21:15:49 (1789607749) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446827 s, 23.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8439] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 21:15:52 (1789607752) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 21:16:01 (1789607761) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 21:16:09 (1789607769) -rw-r--r-- 1 root root 0 Sep 16 21:16 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 21:16:16 (1789607776) -rw-r--r-- 1 root root 0 Sep 16 21:16 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 21:16:23 (1789607783) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 21:16:30 (1789607790) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 21:16:37 (1789607797) total 1 -rwxrwxrwx 1 root root 0 Sep 16 21:16 f2a -rwxrwxrwx 1 root root 0 Sep 16 21:16 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 16 21:16 f2a -rwxrwxrwx 1 root root 0 Sep 16 21:16 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0202115 s, 25.3 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 21:16:44 (1789607804) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (7s) == sanityn test 2g: check blocks update on sync write ==== 21:16:52 (1789607812) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.331092 s, 6.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 21:16:59 (1789607819) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 21:17:06 (1789607826) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 21:17:14 (1789607834) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 21:17:20 (1789607840) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 21:17:28 (1789607848) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 21:17:35 (1789607855) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 21:17:41 (1789607861) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 21:17:49 (1789607869) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889783 s, 11.8 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0971735 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 21:17:57 (1789607877) 1+0 records in 1+0 records out 1 byte copied, 0.00843192 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0138934 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0166446 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00953037 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0112096 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.012712 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0130062 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00978175 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0100752 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0159687 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102794 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00903324 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 21:18:05 (1789607885) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0270538 s, 114 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0554775 s, 73.8 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000272504 s, 11.3 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 21:18:12 (1789607892) striped dir -i1 -c2 -H all_char /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 21:18:21 (1789607901) warning: '-runtime' deprecated, use '-t runtime' instead - link 336 (time 1789607913.68 total 10.01 last 33.57) - link 638 (time 1789607923.70 total 20.03 last 30.13) - link 920 (time 1789607933.72 total 30.05 last 28.14) - link 1215 (time 1789607943.73 total 40.06 last 29.49) - link 1544 (time 1789607953.75 total 50.08 last 32.82) - link 1838 (time 1789607963.76 total 60.10 last 29.36) - link 2103 (time 1789607973.77 total 70.11 last 26.48) - link 2367 (time 1789607983.80 total 80.13 last 26.34) - link 2632 (time 1789607993.81 total 90.14 last 26.46) total: 2919 link in 100.03 seconds: 29.18 ops/second using seed 952520125 running for 100 seconds - stat 7075 (time 1789607915 ; total 11 ; last 11) - stat 10000 (time 1789607919 ; total 15 ; last 4) - stat 16256 (time 1789607930 ; total 26 ; last 11) - stat 20000 (time 1789607935 ; total 31 ; last 5) - stat 27040 (time 1789607946 ; total 42 ; last 11) - stat 30000 (time 1789607949 ; total 45 ; last 3) - stat 35294 (time 1789607960 ; total 56 ; last 11) - stat 38847 (time 1789607971 ; total 67 ; last 11) - stat 40000 (time 1789607973 ; total 69 ; last 2) - stat 43167 (time 1789607984 ; total 80 ; last 11) - stat 47004 (time 1789607995 ; total 91 ; last 11) - stat 50000 (time 1789608003 ; total 99 ; last 8) total: 50056 stats in 100 seconds: 500.559998 stats/second - unlinked 0 (time 1789607954 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2919) error: No such file or directory total: 2918 unlinks in 82 seconds: 35.585365 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22813) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22815) - No such process PASS 12 (141s) == sanityn test 13: test directory page revocation ======= 21:20:42 (1789608042) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d13 f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 21:20:51 (1789608051) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 24024 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 21:20:59 (1789608059) striped dir -i0 -c2 -H crush /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 24615 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (9s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 21:21:08 (1789608068) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 25205 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 21:21:16 (1789608076) striped dir -i0 -c2 -H all_char /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 25800 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 21:21:25 (1789608085) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 26395 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 21:21:35 (1789608095) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7530496 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7530496KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (59s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 21:22:34 (1789608154) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.772207 s, 10.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.615213 s, 13.6 MB/s Chance of close/open is 1 in 50 Seed set to 8160 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero length punch hole truncating to largest ever: 0x2ebace 000006[1] 1789608162.024980 trunc from 00000000 to 0x2ebacd (0x2ebace bytes) 000007[1] 1789608162.102115 write 0x228dff thru 0x22ff0d (0x710f bytes) 000008[1] 1789608162.197836 read 0x0e3c66 thru 0x0f1e5c (0xe1f7 bytes) 000009[1] 1789608162.319432 mapwrite 0x46d394 thru 0x47a4c8 (0xd135 bytes) truncating to largest ever: 0x7e2fd1 000013[0] 1789608162.870636 trunc from 0x47a4c9 to 0x7e2fd0 (0x368b08 bytes) 000015[1] 1789608162.972380 read 0x45686b thru 0x456eea (0x680 bytes) 000017[0] 1789608163.037090 mapread 0x36834b thru 0x371609 (0x92bf bytes) 000018[0] 1789608163.094426 write 0x693ecd thru 0x69f650 (0xb784 bytes) 000019[0] 1789608163.151516 trunc from 0x7e2fd1 to 0x209251 (0x5d9d7f bytes) 000021[1] 1789608163.181298 mapread 0x01b5fc thru 0x02051f (0x4f24 bytes) 000022[1] 1789608163.252907 mapread 0x0f5cac thru 0x0feae0 (0x8e35 bytes) 000023[0] 1789608163.349994 trunc from 0x209252 to 0x2b20f5 (0xa8ea4 bytes) 000024[0] 1789608163.411424 punch from 0x1b77da to 0x1c0837 (0x905e bytes) 000027[1] 1789608163.457079 trunc from 0x2b20f6 to 0x256cde (0x5b417 bytes) 000030[0] 1789608163.593494 punch from 0x09e222 to 0x0ab5d8 (0xd3b7 bytes) 000033[0] 1789608163.686861 mapread 0x03f265 thru 0x047542 (0x82de bytes) 000035[0] 1789608163.778227 mapwrite 0x781e7a thru 0x78eebf (0xd046 bytes) 000038[1] 1789608164.127222 mapread 0x6b263a thru 0x6c0f64 (0xe92b bytes) 000040[1] 1789608164.314280 mapwrite 0x28413e thru 0x2870ec (0x2faf bytes) 000041[1] 1789608164.510438 trunc from 0x78eec0 to 0x62089d (0x16e622 bytes) truncating to largest ever: 0x90cff0 000042[1] 1789608164.616113 trunc from 0x62089e to 0x90cfef (0x2ec752 bytes) 000044[1] 1789608164.736003 mapread 0x811587 thru 0x81c3c3 (0xae3d bytes) 000045[0] 1789608164.872317 trunc from 0x90cff0 to 0x4a31eb (0x469e04 bytes) 000046[0] 1789608164.967076 trunc from 0x4a31ec to 0x860342 (0x3bd157 bytes) 000047[1] 1789608165.098210 read 0x3f8876 thru 0x407eb9 (0xf644 bytes) 000049[1] 1789608165.257296 read 0x414a82 thru 0x423418 (0xe997 bytes) 000052[0] 1789608165.324329 read 0x4e2958 thru 0x4ea2c5 (0x796e bytes) 000053[1] 1789608165.341176 write 0x9e4297 thru 0x9e941a (0x5184 bytes) 000054[0] 1789608165.376779 mapread 0x13852b thru 0x143903 (0xb3d9 bytes) 000058[1] 1789608165.540270 mapwrite 0x132aa6 thru 0x13e98f (0xbeea bytes) 000059[0] 1789608165.723899 write 0x29ea0f thru 0x2aae35 (0xc427 bytes) 000060[1] 1789608165.799376 write 0x855edc thru 0x8610e5 (0xb20a bytes) 000061[0] 1789608165.913468 mapread 0x24b5d0 thru 0x25053c (0x4f6d bytes) 000064[0] 1789608166.020440 write 0x307510 thru 0x316001 (0xeaf2 bytes) 000065[0] 1789608166.073427 punch from 0x4649d1 to 0x46e7b3 (0x9de3 bytes) 000067[1] 1789608166.195420 read 0x3c4cd1 thru 0x3cfcb5 (0xafe5 bytes) 000071[0] 1789608166.292751 write 0x319c70 thru 0x3207e0 (0x6b71 bytes) 000072[0] 1789608166.323700 trunc from 0x9e941b to 0x52f554 (0x4b9ec6 bytes) 000075[0] 1789608166.383287 read 0x4d90d4 thru 0x4db29b (0x21c8 bytes) 000076[1] 1789608166.417723 trunc from 0x52f555 to 0x686b87 (0x157633 bytes) 000077[0] 1789608166.506696 write 0x38f8d8 thru 0x39d090 (0xd7b9 bytes) 000079[0] 1789608166.632708 trunc from 0x686b88 to 0x4e015c (0x1a6a2b bytes) 000080[1] 1789608166.723879 trunc from 0x4e015d to 0x003535 (0x4dcc27 bytes) 000081[0] 1789608166.835486 punch from 0x0031d2 to 0x003535 (0x364 bytes) 000082[0] 1789608166.912410 write 0x1f3902 thru 0x1f5faf (0x26ae bytes) 000084[0] 1789608167.012102 write 0x4aaf72 thru 0x4b24dc (0x756b bytes) 000085[0] 1789608167.042487 read 0x185716 thru 0x187b8a (0x2475 bytes) 000086[1] 1789608167.085901 punch from 0x3c8ec7 to 0x3d0d34 (0x7e6e bytes) 000087[1] 1789608167.164195 read 0x19f37e thru 0x1ac767 (0xd3ea bytes) 000089[0] 1789608167.246196 trunc from 0x4b24dd to 0x0122fc (0x4a01e0 bytes) 000090[0] 1789608167.297086 mapread 0x00e8b5 thru 0x0122fc (0x3a48 bytes) 000091[0] 1789608167.394435 read 0x008214 thru 0x0122fc (0xa0e9 bytes) 000092[1] 1789608167.443770 trunc from 0x0122fd to 0x299e49 (0x287b4d bytes) 000096[1] 1789608167.607020 write 0x861849 thru 0x86dd95 (0xc54d bytes) 000097[1] 1789608167.614039 write 0x7c2535 thru 0x7c7b38 (0x5604 bytes) 000098[0] 1789608167.638880 punch from 0x013610 to 0x02305f (0xfa50 bytes) 000099[1] 1789608167.760213 trunc from 0x86dd96 to 0x41d5d9 (0x4507bc bytes) 000106[1] 1789608167.914230 write 0x4f03ca thru 0x4ffc2c (0xf863 bytes) 000107[0] 1789608167.982502 trunc from 0x4ffc2d to 0x22705e (0x2d8bce bytes) 000109[0] 1789608168.129148 punch from 0x0d0865 to 0x0df497 (0xec33 bytes) 000111[0] 1789608168.200821 punch from 0x0bb8c5 to 0x0c27b1 (0x6eed bytes) 000112[1] 1789608168.207263 punch from 0x0a18f1 to 0x0a64c9 (0x4bd9 bytes) 000114[1] 1789608168.281186 mapwrite 0x96fd47 thru 0x97ee9f (0xf159 bytes) 000115[0] 1789608168.647142 write 0x988daa thru 0x997aeb (0xed42 bytes) 000116[0] 1789608168.688401 mapread 0x8b3e25 thru 0x8bf7f1 (0xb9cd bytes) 000118[0] 1789608169.084850 mapread 0x024f6f thru 0x02eb87 (0x9c19 bytes) 000120[0] 1789608169.189862 punch from 0x71f095 to 0x72c428 (0xd394 bytes) 000121[1] 1789608169.202332 mapwrite 0x750863 thru 0x754635 (0x3dd3 bytes) 000123[0] 1789608169.403364 write 0x483101 thru 0x483b5a (0xa5a bytes) 000125[0] 1789608169.661916 trunc from 0x997aec to 0x1412aa (0x856841 bytes) 000126[0] 1789608169.728460 write 0x5b1f10 thru 0x5bde86 (0xbf77 bytes) 000128[1] 1789608169.807242 mapwrite 0x720ff3 thru 0x72b55f (0xa56d bytes) 000129[0] 1789608170.125181 read 0x6c26b4 thru 0x6c3542 (0xe8f bytes) 000131[1] 1789608170.231986 trunc from 0x72b560 to 0x90c4ff (0x1e0fa0 bytes) 000132[0] 1789608170.299097 mapread 0x063239 thru 0x06a0a3 (0x6e6b bytes) 000133[1] 1789608170.382139 punch from 0x0ad76b to 0x0adea8 (0x73e bytes) 000135[1] 1789608170.439991 mapwrite 0x33a9cc thru 0x349b9a (0xf1cf bytes) 000136[0] 1789608170.527554 read 0x559ef5 thru 0x563dbf (0x9ecb bytes) 000137[0] 1789608170.569020 trunc from 0x90c500 to 0x7e1dad (0x12a752 bytes) 000138[1] 1789608170.636437 write 0x182eb5 thru 0x18a76f (0x78bb bytes) 000139[0] 1789608170.686444 punch from 0x09b5a1 to 0x0a0ca1 (0x5701 bytes) 000140[0] 1789608170.754427 punch from 0x17eb71 to 0x18b9ca (0xce5a bytes) 000141[1] 1789608170.798111 read 0x7d64e1 thru 0x7dd569 (0x7089 bytes) 000143[0] 1789608170.848216 trunc from 0x7e1dae to 0x5c7a60 (0x21a34d bytes) 000144[1] 1789608170.921001 read 0x0d105a thru 0x0d70ef (0x6096 bytes) 000146[1] 1789608170.997127 punch from 0x393ac1 to 0x3a1f7c (0xe4bc bytes) 000149[0] 1789608171.113208 write 0x68015a thru 0x682cd6 (0x2b7d bytes) 000150[0] 1789608171.137118 trunc from 0x682cd7 to 0x47128d (0x211a49 bytes) 000151[0] 1789608171.221198 read 0x1763e9 thru 0x17cfb5 (0x6bcd bytes) 000152[0] 1789608171.266542 write 0x059d42 thru 0x05c67f (0x293e bytes) 000155[0] 1789608171.358464 trunc from 0x47128e to 0x6bacc8 (0x249a3b bytes) 000156[1] 1789608171.405794 punch from 0x5bbb3d to 0x5c6f4a (0xb40e bytes) 000157[0] 1789608171.461093 mapwrite 0x2c45c5 thru 0x2d3847 (0xf283 bytes) 000158[1] 1789608171.663632 punch from 0x5d5648 to 0x5d7886 (0x223f bytes) 000159[1] 1789608171.714452 trunc from 0x6bacc9 to 0x686a3e (0x3428a bytes) 000160[1] 1789608171.859024 punch from 0x3de8ac to 0x3ebebf (0xd614 bytes) 000162[0] 1789608171.940757 mapread 0x2e355c thru 0x2f21a2 (0xec47 bytes) 000163[0] 1789608172.098268 read 0x5bfdd9 thru 0x5c743a (0x7662 bytes) 000165[1] 1789608172.293307 read 0x172263 thru 0x17518d (0x2f2b bytes) 000166[1] 1789608172.333063 mapwrite 0x96129a thru 0x9624bb (0x1222 bytes) 000169[0] 1789608172.645352 read 0x194ca6 thru 0x19b183 (0x64de bytes) 000171[0] 1789608172.683445 mapwrite 0x0d72aa thru 0x0d941e (0x2175 bytes) 000172[0] 1789608172.863627 read 0x7fa59e thru 0x806fe3 (0xca46 bytes) 000173[1] 1789608172.976436 trunc from 0x9624bc to 0x02c3f0 (0x9360cb bytes) 000174[0] 1789608173.094081 mapread 0x01cb6a thru 0x01f5cf (0x2a66 bytes) 000175[1] 1789608173.159065 punch from 0x0107c1 to 0x017b71 (0x73b1 bytes) 000176[0] 1789608173.187596 mapread 0x021b6c thru 0x02c3f0 (0xa885 bytes) 000178[0] 1789608173.291999 mapread 0x025f64 thru 0x0295ab (0x3648 bytes) 000179[0] 1789608173.308199 read 0x026caf thru 0x028c50 (0x1fa2 bytes) 000180[0] 1789608173.309020 mapread 0x015fc7 thru 0x01d6e2 (0x771c bytes) 000184[0] 1789608173.361123 read 0x0041f5 thru 0x008b08 (0x4914 bytes) 000185[0] 1789608173.380386 trunc from 0x02c3f1 to 0x843e8d (0x817a9d bytes) 000186[0] 1789608173.510557 write 0x9219e4 thru 0x92b7a3 (0x9dc0 bytes) 000187[0] 1789608173.513809 write 0x578a7b thru 0x588098 (0xf61e bytes) 000188[0] 1789608173.681412 mapwrite 0x9c3b0c thru 0x9ce452 (0xa947 bytes) 000190[1] 1789608173.857340 punch from 0x603d03 to 0x60ce8f (0x918d bytes) 000191[0] 1789608173.922110 read 0x6b7515 thru 0x6be460 (0x6f4c bytes) 000192[0] 1789608173.974866 mapread 0x78ea3d thru 0x7945b7 (0x5b7b bytes) 000195[0] 1789608174.079052 mapwrite 0x1f8cfe thru 0x1fc4c5 (0x37c8 bytes) 000197[1] 1789608174.354217 mapwrite 0x308d59 thru 0x318336 (0xf5de bytes) 000198[0] 1789608174.746114 write 0x6bc26d thru 0x6be7b2 (0x2546 bytes) 000199[0] 1789608174.839792 read 0x80ac01 thru 0x817630 (0xca30 bytes) 000200[0] 1789608174.889670 read 0x839cb1 thru 0x843e11 (0xa161 bytes) 000202[1] 1789608174.934431 trunc from 0x9ce453 to 0x7a10e8 (0x22d36a bytes) 000203[1] 1789608174.997174 write 0x8d9d8c thru 0x8de145 (0x43ba bytes) truncating to largest ever: 0x9a7e16 000204[0] 1789608175.000508 trunc from 0x8de146 to 0x9a7e15 (0xc9cd0 bytes) 000205[1] 1789608175.090113 mapread 0x310fba thru 0x31719b (0x61e2 bytes) 000206[1] 1789608175.170114 read 0x69503d thru 0x69cb2c (0x7af0 bytes) 000207[1] 1789608175.202064 punch from 0x99ce96 to 0x9a1b8f (0x4cfa bytes) 000208[1] 1789608175.224181 trunc from 0x9a7e16 to 0x49f3ac (0x508a69 bytes) 000209[1] 1789608175.357076 mapread 0x0cdd39 thru 0x0d251b (0x47e3 bytes) 000212[0] 1789608175.452727 mapread 0x1456b1 thru 0x14fe59 (0xa7a9 bytes) 000213[1] 1789608175.538862 trunc from 0x49f3ad to 0x037880 (0x467b2c bytes) 000215[0] 1789608175.661285 write 0x4991c1 thru 0x49f0b2 (0x5ef2 bytes) 000216[1] 1789608175.733391 mapwrite 0x4d9c6e thru 0x4e34d4 (0x9867 bytes) 000217[1] 1789608176.216190 mapwrite 0x297ee9 thru 0x2a3f37 (0xc04f bytes) 000221[1] 1789608176.549927 mapwrite 0x62cef7 thru 0x6348ff (0x7a09 bytes) 000222[0] 1789608176.791079 punch from 0x2a2993 to 0x2ad0ab (0xa719 bytes) 000223[1] 1789608176.856179 mapread 0x09df8b thru 0x0adc81 (0xfcf7 bytes) 000225[1] 1789608176.975061 read 0x5aa418 thru 0x5ac94b (0x2534 bytes) 000226[0] 1789608176.982902 mapwrite 0x3f8c58 thru 0x406c7d (0xe026 bytes) 000227[1] 1789608177.299992 mapread 0x0751fb thru 0x082dca (0xdbd0 bytes) 000228[1] 1789608177.417679 punch from 0x1a3e34 to 0x1acfdd (0x91aa bytes) 000229[1] 1789608177.454969 write 0x4ce123 thru 0x4d7a54 (0x9932 bytes) 000230[0] 1789608177.536739 punch from 0x454546 to 0x46135f (0xce1a bytes) 000231[1] 1789608177.586775 read 0x1f521b thru 0x1f935e (0x4144 bytes) 000232[1] 1789608177.617196 mapwrite 0x4cbaaa thru 0x4d6e2f (0xb386 bytes) 000233[1] 1789608177.725192 trunc from 0x634900 to 0x6631ec (0x2e8ed bytes) 000237[1] 1789608177.894314 read 0x289ce9 thru 0x2973c5 (0xd6dd bytes) 000238[0] 1789608177.938778 trunc from 0x6631ed to 0x5006b8 (0x162b34 bytes) 000239[1] 1789608178.000162 punch from 0x39cc23 to 0x3a960f (0xc9ed bytes) 000240[1] 1789608178.051315 read 0x23cbb5 thru 0x24bafb (0xef47 bytes) 000241[0] 1789608178.093318 mapread 0x1abdeb thru 0x1b3d1c (0x7f32 bytes) 000244[0] 1789608178.227069 trunc from 0x5006b9 to 0x206e6e (0x2f984a bytes) 000245[0] 1789608178.307994 mapwrite 0x76851f thru 0x772bcd (0xa6af bytes) 000246[1] 1789608178.619153 read 0x402d40 thru 0x404314 (0x15d5 bytes) 000247[1] 1789608178.653046 punch from 0x224e69 to 0x22b988 (0x6b20 bytes) truncating to largest ever: 0x9d3e89 000248[0] 1789608178.691996 trunc from 0x772bce to 0x9d3e88 (0x2612bb bytes) 000252[1] 1789608178.804391 trunc from 0x9d3e89 to 0x5ec869 (0x3e761f bytes) 000253[0] 1789608178.925609 mapwrite 0x9ef6a1 thru 0x9efec8 (0x828 bytes) 000254[1] 1789608179.125636 trunc from 0x9efec9 to 0x26fd0c (0x7801bc bytes) 000256[0] 1789608179.261093 trunc from 0x26fd0d to 0x893910 (0x623c04 bytes) 000257[0] 1789608179.394716 mapwrite 0x87b379 thru 0x87dce1 (0x2969 bytes) 000258[0] 1789608179.642385 mapread 0x16c95d thru 0x172dec (0x6490 bytes) 000259[1] 1789608179.741921 mapread 0x7555b5 thru 0x761a4a (0xc496 bytes) 000260[1] 1789608179.855200 trunc from 0x893911 to 0x60d1aa (0x286766 bytes) 000262[1] 1789608180.015341 mapread 0x2a9367 thru 0x2aaf80 (0x1c1a bytes) 000263[0] 1789608180.080256 mapread 0x28754a thru 0x293d33 (0xc7ea bytes) 000264[0] 1789608180.227193 trunc from 0x60d1ab to 0x562295 (0xaaf15 bytes) 000265[0] 1789608180.304973 read 0x227ccd thru 0x22d7bb (0x5aef bytes) 000267[1] 1789608180.376067 mapwrite 0x903331 thru 0x906380 (0x3050 bytes) 000268[1] 1789608180.748805 punch from 0x65239c to 0x6570e9 (0x4d4e bytes) 000269[1] 1789608180.815720 punch from 0x7d62ff to 0x7e413b (0xde3d bytes) 000271[0] 1789608180.859919 mapread 0x511f1c thru 0x519495 (0x757a bytes) 000272[1] 1789608180.925043 mapread 0x40edae thru 0x41c3d0 (0xd623 bytes) 000275[0] 1789608180.993400 punch from 0x625cde to 0x63077a (0xaa9d bytes) 000276[0] 1789608181.059036 trunc from 0x906381 to 0x77a330 (0x18c050 bytes) 000277[0] 1789608181.111310 mapread 0x277929 thru 0x286432 (0xeb0a bytes) 000278[0] 1789608181.177199 trunc from 0x77a331 to 0x0012fe (0x779032 bytes) 000279[0] 1789608181.239035 punch from 0x00080d to 0x0012fe (0xaf2 bytes) 000280[0] 1789608181.312336 punch from 0x000682 to 0x0012fe (0xc7d bytes) 000281[0] 1789608181.370098 read 0x0002d1 thru 0x0012fe (0x102e bytes) 000282[0] 1789608181.399416 trunc from 0x0012ff to 0x508b30 (0x507832 bytes) 000284[0] 1789608181.619128 punch from 0x015a11 to 0x01f485 (0x9a75 bytes) 000285[1] 1789608181.644158 mapwrite 0x43a1c0 thru 0x443e40 (0x9c81 bytes) 000287[1] 1789608181.853741 punch from 0x0d9325 to 0x0dff21 (0x6bfd bytes) 000288[0] 1789608181.875679 punch from 0x4c3ba1 to 0x4c754b (0x39ab bytes) 000289[1] 1789608181.926530 write 0x42b725 thru 0x439836 (0xe112 bytes) 000291[1] 1789608182.023222 mapwrite 0x1f45c4 thru 0x1f6f87 (0x29c4 bytes) 000294[1] 1789608182.222301 trunc from 0x508b31 to 0x4b21b9 (0x56977 bytes) 000295[0] 1789608182.268610 write 0x63f533 thru 0x644313 (0x4de1 bytes) 000297[0] 1789608182.386812 write 0x9f7464 thru 0x9fffff (0x8b9c bytes) 000298[0] 1789608182.419377 mapwrite 0x31bfbb thru 0x32a33f (0xe385 bytes) 000299[1] 1789608182.592320 write 0x90c661 thru 0x91465b (0x7ffb bytes) 000301[0] 1789608182.667485 mapwrite 0x1a0a10 thru 0x1a6ca3 (0x6294 bytes) 000303[0] 1789608182.822130 mapwrite 0x25e0f1 thru 0x268825 (0xa735 bytes) 000307[1] 1789608183.067744 trunc from 0xa00000 to 0x7f2e61 (0x20d19e bytes) 000308[1] 1789608183.163232 read 0x312fac thru 0x314531 (0x1586 bytes) 000309[1] 1789608183.198102 read 0x7ad2de thru 0x7ae2d0 (0xff3 bytes) 000310[1] 1789608183.228058 punch from 0x5f2c52 to 0x5f8743 (0x5af2 bytes) 000312[0] 1789608183.266262 mapread 0x682f7e thru 0x6917b2 (0xe835 bytes) 000313[1] 1789608183.397979 trunc from 0x7f2e62 to 0x709ce8 (0xe9179 bytes) 000314[1] 1789608183.465103 write 0x0c2854 thru 0x0cefb2 (0xc75f bytes) 000315[0] 1789608183.481186 punch from 0x6726dc to 0x68090b (0xe230 bytes) 000316[1] 1789608183.536305 read 0x286d64 thru 0x28e649 (0x78e6 bytes) 000317[1] 1789608183.601740 read 0x23916a thru 0x23b81b (0x26b2 bytes) 000323[0] 1789608183.796187 punch from 0x61c36c to 0x62a2b1 (0xdf46 bytes) 000324[0] 1789608183.828522 trunc from 0x709ce9 to 0x72ee59 (0x25171 bytes) 000325[1] 1789608183.939083 mapwrite 0x3e0e63 thru 0x3efebc (0xf05a bytes) 000326[0] 1789608184.224979 mapwrite 0x45c5cb thru 0x4649d4 (0x840a bytes) 000327[0] 1789608184.401177 read 0x67bc19 thru 0x67c6c6 (0xaae bytes) 000328[0] 1789608184.437171 trunc from 0x72ee5a to 0x97f319 (0x2504c0 bytes) 000329[0] 1789608184.532819 mapwrite 0x85c6bd thru 0x860365 (0x3ca9 bytes) 000330[1] 1789608184.636267 punch from 0x31a2a1 to 0x325987 (0xb6e7 bytes) 000331[0] 1789608184.676074 mapwrite 0x8881cc thru 0x88bb5f (0x3994 bytes) 000334[1] 1789608184.793356 read 0x36b8b7 thru 0x372132 (0x687c bytes) 000335[1] 1789608184.826161 mapread 0x3f8038 thru 0x3f9a98 (0x1a61 bytes) 000336[0] 1789608184.883550 write 0x46e91f thru 0x471df6 (0x34d8 bytes) 000337[1] 1789608184.926605 read 0x965a5f thru 0x970df9 (0xb39b bytes) 000338[0] 1789608184.937168 read 0x3a09d2 thru 0x3a3b2d (0x315c bytes) 000339[0] 1789608184.962643 read 0x6879f3 thru 0x689587 (0x1b95 bytes) 000342[1] 1789608185.017075 mapwrite 0x6e82b6 thru 0x6e96af (0x13fa bytes) 000345[0] 1789608185.168691 mapwrite 0x4134b7 thru 0x41a7ac (0x72f6 bytes) 000346[1] 1789608185.357139 read 0x42981c thru 0x42bb96 (0x237b bytes) 000347[1] 1789608185.410942 trunc from 0x97f31a to 0x906525 (0x78df4 bytes) 000348[0] 1789608185.552304 trunc from 0x906526 to 0x341f9b (0x5c458a bytes) 000349[0] 1789608185.623742 write 0x3881d2 thru 0x391723 (0x9552 bytes) 000350[0] 1789608185.635736 mapread 0x245a19 thru 0x24a68f (0x4c77 bytes) 000351[0] 1789608185.659337 read 0x304614 thru 0x313757 (0xf144 bytes) 000352[0] 1789608185.753179 read 0x0bee81 thru 0x0c9531 (0xa6b1 bytes) 000353[1] 1789608185.755923 write 0x7c6075 thru 0x7c73bd (0x1349 bytes) 000357[1] 1789608185.927690 punch from 0x291242 to 0x2916db (0x49a bytes) 000358[1] 1789608186.015882 mapread 0x722e67 thru 0x72dd60 (0xaefa bytes) 000361[0] 1789608186.069535 mapread 0x177130 thru 0x182812 (0xb6e3 bytes) 000362[0] 1789608186.205591 read 0x1b2e5e thru 0x1c096c (0xdb0f bytes) 000363[0] 1789608186.206335 mapwrite 0x1545aa thru 0x15a3c7 (0x5e1e bytes) 000364[0] 1789608186.492347 write 0x9adedd thru 0x9bce63 (0xef87 bytes) 000365[1] 1789608186.546244 punch from 0x870938 to 0x877285 (0x694e bytes) 000366[0] 1789608186.636042 mapread 0x19329c thru 0x19af55 (0x7cba bytes) 000369[1] 1789608186.766908 read 0x87e2cf thru 0x888dd5 (0xab07 bytes) 000370[1] 1789608186.782304 punch from 0x4493f7 to 0x4541bb (0xadc5 bytes) 000371[1] 1789608186.808034 trunc from 0x9bce64 to 0x7ef0c9 (0x1cdd9a bytes) 000374[1] 1789608186.933216 write 0x7d4336 thru 0x7e104b (0xcd16 bytes) 000375[0] 1789608186.960533 mapwrite 0x25d021 thru 0x26a0bc (0xd09c bytes) 000376[1] 1789608187.199326 punch from 0x5f83f3 to 0x5f88c4 (0x4d2 bytes) 000377[1] 1789608187.258746 read 0x474e5b thru 0x47fb0e (0xacb4 bytes) 000378[1] 1789608187.354195 trunc from 0x7ef0ca to 0x6e25f2 (0x10cad7 bytes) 000379[0] 1789608187.431219 trunc from 0x6e25f3 to 0x5ea36c (0xf8286 bytes) 000382[0] 1789608187.595176 punch from 0x077507 to 0x082333 (0xae2d bytes) 000384[0] 1789608187.644947 write 0x8796ac thru 0x87de34 (0x4789 bytes) 000385[1] 1789608187.650996 punch from 0x1dd88e to 0x1e0500 (0x2c73 bytes) 000386[1] 1789608187.715837 mapwrite 0x8e985a thru 0x8f73e6 (0xdb8d bytes) 000389[1] 1789608188.104848 mapread 0x080803 thru 0x0852c0 (0x4abe bytes) 000393[1] 1789608188.185390 mapwrite 0x8ef6fd thru 0x8fd220 (0xdb24 bytes) 000394[1] 1789608188.518986 punch from 0x8aec8f to 0x8afbe6 (0xf58 bytes) 000395[0] 1789608188.589192 read 0x49de26 thru 0x4ad43e (0xf619 bytes) 000396[0] 1789608188.666023 punch from 0x233d64 to 0x236996 (0x2c33 bytes) 000397[0] 1789608188.692081 read 0x3c88b8 thru 0x3cee97 (0x65e0 bytes) 000399[1] 1789608188.728190 mapwrite 0x7aa7a4 thru 0x7afc8f (0x54ec bytes) 000400[0] 1789608188.869490 punch from 0x320253 to 0x329df7 (0x9ba5 bytes) 000401[1] 1789608188.908835 mapread 0x425922 thru 0x4265d1 (0xcb0 bytes) 000402[1] 1789608188.979128 read 0x605a86 thru 0x60ad38 (0x52b3 bytes) 000403[1] 1789608189.032403 trunc from 0x8fd221 to 0x6fede6 (0x1fe43a bytes) 000404[0] 1789608189.083505 mapwrite 0x8b4c22 thru 0x8b687d (0x1c5c bytes) 000406[0] 1789608189.298626 read 0x1af54d thru 0x1b242e (0x2ee2 bytes) 000407[1] 1789608189.318910 write 0x6c25db thru 0x6d1349 (0xed6f bytes) 000409[0] 1789608189.374148 mapread 0x1cacfa thru 0x1ce4c4 (0x37cb bytes) 000411[0] 1789608189.437348 trunc from 0x8b687e to 0x5b9db2 (0x2fcacb bytes) 000412[1] 1789608189.495480 mapread 0x0b0487 thru 0x0b16e2 (0x125c bytes) 000413[1] 1789608189.579860 read 0x3e0119 thru 0x3ecb23 (0xca0b bytes) 000414[0] 1789608189.628447 trunc from 0x5b9db3 to 0x4d7e76 (0xe1f3c bytes) 000415[1] 1789608189.692230 read 0x34d39c thru 0x34dda2 (0xa07 bytes) 000417[1] 1789608189.757000 punch from 0x4d01dc to 0x4d7e76 (0x7c9b bytes) 000419[0] 1789608189.857654 punch from 0x2df83e to 0x2ef445 (0xfc08 bytes) 000420[1] 1789608189.925998 read 0x0c84bb thru 0x0d27d0 (0xa316 bytes) 000421[0] 1789608189.996132 read 0x123dba thru 0x126c65 (0x2eac bytes) 000424[1] 1789608190.076131 write 0x860a5a thru 0x86e19b (0xd742 bytes) 000426[1] 1789608190.109995 mapread 0x7b99cf thru 0x7b9cf2 (0x324 bytes) 000427[1] 1789608190.115829 mapwrite 0x14ed54 thru 0x158841 (0x9aee bytes) 000428[1] 1789608190.300290 read 0x70026a thru 0x700cdf (0xa76 bytes) 000429[1] 1789608190.344524 write 0x3a093a thru 0x3a31bf (0x2886 bytes) 000430[0] 1789608190.404369 mapread 0x7e3a1f thru 0x7e6679 (0x2c5b bytes) 000432[0] 1789608190.535141 read 0x7ee4e8 thru 0x7f9dce (0xb8e7 bytes) 000433[0] 1789608190.641033 trunc from 0x86e19c to 0x66c7d3 (0x2019c8 bytes) 000434[0] 1789608190.708818 read 0x5cf230 thru 0x5d60d2 (0x6ea3 bytes) 000435[1] 1789608190.720213 mapwrite 0x50ae24 thru 0x5167bb (0xb998 bytes) 000438[0] 1789608190.995092 read 0x0c3938 thru 0x0c55f6 (0x1cbf bytes) 000439[0] 1789608191.018422 mapwrite 0x522de2 thru 0x525215 (0x2434 bytes) 000440[1] 1789608191.212757 trunc from 0x66c7d4 to 0x110ef7 (0x55b8dc bytes) 000441[0] 1789608191.389968 read 0x045f23 thru 0x04c30a (0x63e8 bytes) 000443[1] 1789608191.549231 mapwrite 0x988605 thru 0x992a0c (0xa408 bytes) 000445[0] 1789608191.933751 punch from 0x1746a1 to 0x17b3da (0x6d3a bytes) 000448[1] 1789608192.087421 mapwrite 0x72211f thru 0x7240c6 (0x1fa8 bytes) 000449[1] 1789608192.180240 write 0x27321e thru 0x27a88b (0x766e bytes) 000450[1] 1789608192.210968 trunc from 0x992a0d to 0x57a89a (0x418172 bytes) 000451[1] 1789608192.278628 trunc from 0x57a89b to 0x10c3b9 (0x46e4e1 bytes) 000452[1] 1789608192.340415 trunc from 0x10c3ba to 0x316f54 (0x20ab9b bytes) 000453[0] 1789608192.398373 mapread 0x06963e thru 0x06f428 (0x5deb bytes) 000454[1] 1789608192.524131 write 0x18717d thru 0x192e9b (0xbd1f bytes) 000455[1] 1789608192.587181 write 0x4b3914 thru 0x4b464a (0xd37 bytes) 000457[1] 1789608192.620223 punch from 0x3510b1 to 0x35789e (0x67ee bytes) 000458[0] 1789608192.626185 read 0x07435c thru 0x0833b4 (0xf059 bytes) 000460[0] 1789608192.701183 write 0x8bc9c1 thru 0x8c19f4 (0x5034 bytes) 000461[1] 1789608192.741315 mapwrite 0x11f633 thru 0x127882 (0x8250 bytes) 000463[0] 1789608192.904121 mapread 0x352a63 thru 0x35dc22 (0xb1c0 bytes) 000465[0] 1789608193.063941 trunc from 0x8c19f5 to 0x18e6c6 (0x73332e bytes) 000466[1] 1789608193.193787 punch from 0x0cc836 to 0x0d96d9 (0xcea4 bytes) 000467[0] 1789608193.261646 mapwrite 0x75dc2d thru 0x7609e8 (0x2dbc bytes) 000468[0] 1789608193.397050 mapwrite 0x6d2294 thru 0x6d77e3 (0x5550 bytes) 000469[1] 1789608193.544914 mapread 0x293612 thru 0x295be1 (0x25d0 bytes) 000470[0] 1789608193.668110 write 0x569bb1 thru 0x56ae78 (0x12c8 bytes) 000471[1] 1789608193.727536 mapwrite 0x7ce4d2 thru 0x7ceb5e (0x68d bytes) 000472[0] 1789608194.074212 read 0x171327 thru 0x17ad90 (0x9a6a bytes) 000473[0] 1789608194.109176 read 0x393489 thru 0x395a2b (0x25a3 bytes) 000475[1] 1789608194.334177 mapread 0x2c0ab6 thru 0x2c91c4 (0x870f bytes) 000476[0] 1789608194.388766 read 0x0b7c1e thru 0x0bad4d (0x3130 bytes) 000477[0] 1789608194.389270 write 0x22bbc7 thru 0x237996 (0xbdd0 bytes) 000482[1] 1789608194.663254 punch from 0x40e4e7 to 0x415f04 (0x7a1e bytes) 000485[0] 1789608194.868459 punch from 0x211c26 to 0x218d51 (0x712c bytes) 000486[0] 1789608194.914436 trunc from 0x7ceb5f to 0x0f875a (0x6d6404 bytes) 000488[1] 1789608195.007396 mapwrite 0x7e1df7 thru 0x7f0bbe (0xedc8 bytes) 000490[1] 1789608195.240138 write 0x7fa91b thru 0x808726 (0xde0c bytes) 000492[1] 1789608195.271504 write 0x7c4aaa thru 0x7d3a9f (0xeff6 bytes) 000493[1] 1789608195.317055 mapread 0x39e13a thru 0x3a68b0 (0x8777 bytes) 000494[0] 1789608195.362183 punch from 0x3f363c to 0x3f9d82 (0x6747 bytes) 000495[1] 1789608195.414692 trunc from 0x808727 to 0x3a2f07 (0x46581f bytes) 000498[0] 1789608195.498483 punch from 0x1f39b4 to 0x1f6285 (0x28d2 bytes) 000500[0] 1789608195.549622 punch from 0x1b209e to 0x1b64bf (0x4422 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8195 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1789608196.981748 write_OD 0x85b000 thru 0x867fff (0xd000 bytes) 000004[1] 1789608197.189545 read_OD 0x74f000 thru 0x757fff (0x9000 bytes) 000005[1] 1789608197.220070 mapread 0x2f5000 thru 0x2ffa60 (0xaa61 bytes) 000009[0] 1789608197.257808 punch from 0x8256ef to 0x827e79 (0x278b bytes) 000011[1] 1789608197.317378 mapwrite 0x882000 thru 0x884cd5 (0x2cd6 bytes) 000012[0] 1789608197.460759 mapread 0x20c000 thru 0x20c923 (0x924 bytes) 000014[0] 1789608197.508109 mapread 0x3ae000 thru 0x3b358b (0x558c bytes) 000019[1] 1789608197.536146 write_OD 0x7cf000 thru 0x7d4fff (0x6000 bytes) truncating to largest ever: 0x5f7f08 000020[0] 1789608197.619094 trunc from 0x884cd6 to 0x5f7f07 (0x28cdce bytes) truncating to largest ever: 0x60976c 000021[0] 1789608197.659198 trunc from 0x5f7f08 to 0x60976b (0x11864 bytes) 000022[0] 1789608197.685197 read_OD 0x150000 thru 0x15dfff (0xe000 bytes) 000023[1] 1789608197.718141 trunc from 0x60976c to 0x4ff4c1 (0x10a2aa bytes) 000024[0] 1789608197.796512 write_OD 0x304000 thru 0x30afff (0x7000 bytes) 000025[0] 1789608197.864226 mapread 0x48c000 thru 0x497c3b (0xbc3c bytes) 000026[1] 1789608197.904180 read_OD 0x03f000 thru 0x03ffff (0x1000 bytes) 000027[1] 1789608197.921478 punch from 0x44e744 to 0x454c15 (0x64d2 bytes) 000028[1] 1789608197.972023 mapread 0x4d5000 thru 0x4e2db5 (0xddb6 bytes) 000029[0] 1789608198.008606 write_OD 0x7fc000 thru 0x800fff (0x5000 bytes) 000031[1] 1789608198.101924 trunc from 0x801000 to 0x49b7e3 (0x36581c bytes) 000033[0] 1789608198.181007 write_OD 0x077000 thru 0x085fff (0xf000 bytes) 000034[1] 1789608198.251873 read_OD 0x060000 thru 0x062fff (0x3000 bytes) 000035[0] 1789608198.262829 trunc from 0x49b7e4 to 0x1bc66d (0x2df176 bytes) 000036[1] 1789608198.341280 punch from 0x0b796b to 0x0b9273 (0x1909 bytes) 000037[0] 1789608198.384169 mapread 0x0aa000 thru 0x0ac620 (0x2621 bytes) 000038[0] 1789608198.433948 mapread 0x060000 thru 0x0656e0 (0x56e1 bytes) 000039[0] 1789608198.452118 read_OD 0x0e5000 thru 0x0eefff (0xa000 bytes) 000040[1] 1789608198.461463 mapwrite 0x419000 thru 0x4209c6 (0x79c7 bytes) 000043[1] 1789608198.580400 read_OD 0x073000 thru 0x077fff (0x5000 bytes) 000044[0] 1789608198.612589 write_OD 0x62e000 thru 0x630fff (0x3000 bytes) 000046[0] 1789608198.700746 trunc from 0x631000 to 0x00d805 (0x6237fa bytes) 000048[0] 1789608198.752090 trunc from 0x00d806 to 0x5c5dc3 (0x5b85be bytes) 000051[0] 1789608198.773383 mapread 0x2d8000 thru 0x2e26a9 (0xa6aa bytes) 000053[0] 1789608198.916614 read_OD 0x3ae000 thru 0x3b6fff (0x9000 bytes) 000054[0] 1789608199.011676 write_OD 0x4c5000 thru 0x4cbfff (0x7000 bytes) 000056[0] 1789608199.154958 trunc from 0x5c5dc4 to 0x568d37 (0x5d08c bytes) 000057[1] 1789608199.229808 trunc from 0x568d38 to 0x286d59 (0x2e1fde bytes) 000060[0] 1789608199.309042 mapwrite 0x033000 thru 0x03f53f (0xc540 bytes) 000061[1] 1789608199.568842 read_OD 0x15b000 thru 0x164fff (0xa000 bytes) truncating to largest ever: 0x9dcca4 000062[0] 1789608199.611099 trunc from 0x286d5a to 0x9dcca3 (0x755f4a bytes) 000063[1] 1789608199.689297 trunc from 0x9dcca4 to 0x8bea8b (0x11e218 bytes) 000065[1] 1789608199.766507 read_OD 0x3bc000 thru 0x3c7fff (0xc000 bytes) 000066[0] 1789608199.795842 punch from 0x3dc1eb to 0x3e5943 (0x9759 bytes) 000068[1] 1789608199.869261 mapwrite 0x5f1000 thru 0x5f6260 (0x5261 bytes) 000069[1] 1789608199.999372 write_OD 0x7e5000 thru 0x7f3fff (0xf000 bytes) 000071[0] 1789608200.119878 mapwrite 0x148000 thru 0x156600 (0xe601 bytes) 000073[1] 1789608200.408636 punch from 0x38a000 to 0x38d734 (0x3735 bytes) 000074[0] 1789608200.470348 mapwrite 0x87d000 thru 0x8868d5 (0x98d6 bytes) 000075[1] 1789608200.615126 read_OD 0x680000 thru 0x68cfff (0xd000 bytes) 000076[1] 1789608200.651558 mapwrite 0x164000 thru 0x167585 (0x3586 bytes) 000077[1] 1789608200.763742 mapread 0x08a000 thru 0x08a101 (0x102 bytes) 000079[0] 1789608200.797964 mapwrite 0x774000 thru 0x7776a8 (0x36a9 bytes) 000081[1] 1789608200.989114 write_OD 0x494000 thru 0x4a0fff (0xd000 bytes) 000084[1] 1789608201.114605 punch from 0x77f8c7 to 0x784c4a (0x5384 bytes) 000086[0] 1789608201.165435 read_OD 0x477000 thru 0x47bfff (0x5000 bytes) 000087[0] 1789608201.221473 read_OD 0x455000 thru 0x462fff (0xe000 bytes) 000088[1] 1789608201.237045 trunc from 0x8bea8c to 0x72244b (0x19c640 bytes) 000090[1] 1789608201.314355 write_OD 0x35f000 thru 0x36afff (0xc000 bytes) 000092[0] 1789608201.461307 write_OD 0x806000 thru 0x809fff (0x4000 bytes) 000093[0] 1789608201.535131 mapread 0x355000 thru 0x35ff9b (0xaf9c bytes) 000094[1] 1789608201.565135 punch from 0x000518 to 0x00c0f9 (0xbbe2 bytes) 000095[1] 1789608201.636729 trunc from 0x80a000 to 0x1d0f2b (0x6390d4 bytes) 000096[0] 1789608201.671076 write_OD 0x132000 thru 0x13ffff (0xe000 bytes) 000099[0] 1789608201.842135 mapread 0x0fe000 thru 0x102b9a (0x4b9b bytes) 000100[0] 1789608201.871507 write_OD 0x60e000 thru 0x61bfff (0xe000 bytes) 000101[1] 1789608201.946618 write_OD 0x032000 thru 0x038fff (0x7000 bytes) 000103[0] 1789608202.046103 mapwrite 0x39b000 thru 0x3a101c (0x601d bytes) 000104[0] 1789608202.147160 write_OD 0x750000 thru 0x756fff (0x7000 bytes) 000105[0] 1789608202.213756 punch from 0x5ed04d to 0x5fcebe (0xfe72 bytes) 000107[1] 1789608202.253282 punch from 0x6f50c4 to 0x6fbcf6 (0x6c33 bytes) 000110[0] 1789608202.276675 mapwrite 0x0da000 thru 0x0e89fe (0xe9ff bytes) 000113[0] 1789608202.487305 mapread 0x672000 thru 0x681ca5 (0xfca6 bytes) 000114[1] 1789608202.539522 punch from 0x5e55cc to 0x5f3475 (0xdeaa bytes) 000115[1] 1789608202.572069 mapread 0x2aa000 thru 0x2b7d6b (0xdd6c bytes) 000117[0] 1789608202.726423 mapwrite 0x43b000 thru 0x43b8a2 (0x8a3 bytes) 000119[1] 1789608202.910105 mapwrite 0x647000 thru 0x652d99 (0xbd9a bytes) 000120[1] 1789608203.163044 mapread 0x271000 thru 0x27209c (0x109d bytes) 000121[0] 1789608203.191323 mapread 0x344000 thru 0x34c705 (0x8706 bytes) 000122[0] 1789608203.274514 mapwrite 0x18b000 thru 0x197353 (0xc354 bytes) 000123[0] 1789608203.450052 mapwrite 0x4fd000 thru 0x4fd150 (0x151 bytes) 000124[1] 1789608203.504375 read_OD 0x1ba000 thru 0x1c2fff (0x9000 bytes) 000125[1] 1789608203.535126 mapwrite 0x17d000 thru 0x183daa (0x6dab bytes) 000126[0] 1789608203.666844 mapread 0x5e8000 thru 0x5ef1c9 (0x71ca bytes) 000130[0] 1789608203.776119 trunc from 0x757000 to 0x15f919 (0x5f76e6 bytes) 000132[1] 1789608203.869727 punch from 0x03e477 to 0x04b24a (0xcdd4 bytes) 000133[1] 1789608203.923690 read_OD 0x006000 thru 0x00cfff (0x7000 bytes) 000134[1] 1789608203.952060 read_OD 0x0e0000 thru 0x0e3fff (0x4000 bytes) 000135[1] 1789608203.973993 trunc from 0x15f91a to 0x16c74c (0xce33 bytes) 000137[1] 1789608204.055461 mapread 0x13c000 thru 0x14a93c (0xe93d bytes) 000138[1] 1789608204.100045 punch from 0x0f9a63 to 0x0fa457 (0x9f5 bytes) 000139[0] 1789608204.125644 punch from 0x0a5c9b to 0x0a6725 (0xa8b bytes) 000140[1] 1789608204.152132 read_OD 0x087000 thru 0x095fff (0xf000 bytes) 000142[0] 1789608204.210758 mapread 0x0d8000 thru 0x0d94d5 (0x14d6 bytes) 000143[1] 1789608204.222102 write_OD 0x89f000 thru 0x8a8fff (0xa000 bytes) 000144[1] 1789608204.306947 mapread 0x615000 thru 0x615ef7 (0xef8 bytes) 000146[1] 1789608204.407279 read_OD 0x73a000 thru 0x742fff (0x9000 bytes) 000147[0] 1789608204.475738 mapread 0x550000 thru 0x55f1a5 (0xf1a6 bytes) 000148[1] 1789608204.512764 mapwrite 0x1f8000 thru 0x203664 (0xb665 bytes) 000149[1] 1789608204.694501 read_OD 0x1a1000 thru 0x1abfff (0xb000 bytes) 000151[1] 1789608204.725432 punch from 0x740564 to 0x74b6af (0xb14c bytes) 000154[0] 1789608204.773368 mapread 0x5f1000 thru 0x5f32c4 (0x22c5 bytes) 000155[1] 1789608204.823293 punch from 0x2fd48b to 0x3088ed (0xb463 bytes) 000156[0] 1789608204.861382 mapread 0x464000 thru 0x4680f1 (0x40f2 bytes) 000157[0] 1789608205.041898 write_OD 0x849000 thru 0x850fff (0x8000 bytes) 000158[0] 1789608205.132385 punch from 0x09726a to 0x09b733 (0x44ca bytes) 000159[1] 1789608205.181550 write_OD 0x15d000 thru 0x163fff (0x7000 bytes) 000160[1] 1789608205.242868 mapwrite 0x3cc000 thru 0x3d4f55 (0x8f56 bytes) 000161[0] 1789608205.325646 read_OD 0x17c000 thru 0x182fff (0x7000 bytes) 000164[1] 1789608205.368112 trunc from 0x8a9000 to 0x6389c2 (0x27063d bytes) 000166[0] 1789608205.432267 mapread 0x22a000 thru 0x2372da (0xd2db bytes) 000167[1] 1789608205.511703 mapwrite 0x883000 thru 0x890685 (0xd686 bytes) 000170[0] 1789608205.664306 mapread 0x051000 thru 0x051432 (0x433 bytes) 000172[1] 1789608205.692934 mapread 0x7f6000 thru 0x803ba1 (0xdba2 bytes) 000174[1] 1789608205.719716 trunc from 0x890686 to 0x63a142 (0x256543 bytes) 000175[1] 1789608205.755097 punch from 0x38a43a to 0x399e02 (0xf9c9 bytes) 000176[0] 1789608205.787048 mapread 0x4dc000 thru 0x4dec48 (0x2c49 bytes) 000177[1] 1789608205.839907 read_OD 0x11d000 thru 0x126fff (0xa000 bytes) 000179[1] 1789608205.855091 write_OD 0x88e000 thru 0x898fff (0xb000 bytes) 000180[1] 1789608205.913551 mapread 0x2fe000 thru 0x30c19c (0xe19d bytes) 000181[0] 1789608205.948348 trunc from 0x899000 to 0x0e74df (0x7b1b20 bytes) 000184[0] 1789608206.049426 trunc from 0x0e74e0 to 0x3088ff (0x221420 bytes) 000187[0] 1789608206.090750 punch from 0x0c15d8 to 0x0c624c (0x4c75 bytes) 000188[1] 1789608206.140355 mapwrite 0x751000 thru 0x7512f6 (0x2f7 bytes) 000192[0] 1789608206.286257 read_OD 0x0a6000 thru 0x0aefff (0x9000 bytes) 000199[1] 1789608206.328431 mapread 0x063000 thru 0x068120 (0x5121 bytes) 000201[0] 1789608206.352413 write_OD 0x325000 thru 0x32ffff (0xb000 bytes) 000202[1] 1789608206.408344 mapwrite 0x7e7000 thru 0x7f34b9 (0xc4ba bytes) 000205[0] 1789608206.532754 read_OD 0x0e8000 thru 0x0e8fff (0x1000 bytes) 000206[0] 1789608206.561900 trunc from 0x7f34ba to 0x6bd206 (0x1362b3 bytes) 000208[0] 1789608206.593890 punch from 0x024525 to 0x0263b4 (0x1e90 bytes) 000209[1] 1789608206.613982 write_OD 0x11c000 thru 0x122fff (0x7000 bytes) 000211[1] 1789608206.669016 mapread 0x612000 thru 0x619120 (0x7121 bytes) 000213[1] 1789608206.688488 trunc from 0x6bd207 to 0x42ffdf (0x28d227 bytes) 000214[1] 1789608206.727293 mapread 0x009000 thru 0x00dc2d (0x4c2e bytes) 000215[1] 1789608206.769406 trunc from 0x42ffe0 to 0x111b23 (0x31e4bc bytes) 000216[0] 1789608206.826700 read_OD 0x0d3000 thru 0x0d4fff (0x2000 bytes) 000219[1] 1789608206.876385 mapwrite 0x5a9000 thru 0x5aa61e (0x161f bytes) 000222[1] 1789608207.057169 punch from 0x4f43e4 to 0x4f509d (0xcba bytes) 000223[1] 1789608207.118861 write_OD 0x308000 thru 0x313fff (0xc000 bytes) 000224[1] 1789608207.300826 punch from 0x2ea28f to 0x2f04c8 (0x623a bytes) 000226[1] 1789608207.385417 mapwrite 0x062000 thru 0x0676b9 (0x56ba bytes) 000227[1] 1789608207.501235 mapread 0x089000 thru 0x08efae (0x5faf bytes) 000228[1] 1789608207.591297 read_OD 0x18f000 thru 0x190fff (0x2000 bytes) 000229[1] 1789608207.625143 trunc from 0x5aa61f to 0x758caf (0x1ae691 bytes) 000233[1] 1789608207.703406 read_OD 0x448000 thru 0x44cfff (0x5000 bytes) 000234[1] 1789608207.742991 mapwrite 0x843000 thru 0x84af79 (0x7f7a bytes) 000235[1] 1789608207.861247 write_OD 0x8dc000 thru 0x8e5fff (0xa000 bytes) 000236[1] 1789608207.956521 mapwrite 0x862000 thru 0x869a1e (0x7a1f bytes) 000237[0] 1789608208.075377 mapwrite 0x36d000 thru 0x37bb25 (0xeb26 bytes) 000238[1] 1789608208.265421 mapwrite 0x83d000 thru 0x84bc3f (0xec40 bytes) 000239[0] 1789608208.433845 trunc from 0x8e6000 to 0x6f0be6 (0x1f5419 bytes) 000240[1] 1789608208.479816 mapread 0x585000 thru 0x58c52e (0x752f bytes) 000242[1] 1789608208.556101 punch from 0x6188bc to 0x619b9a (0x12df bytes) 000243[1] 1789608208.614428 trunc from 0x6f0be7 to 0x075007 (0x67bbdf bytes) 000248[0] 1789608208.678366 write_OD 0x4a6000 thru 0x4b3fff (0xe000 bytes) 000249[1] 1789608208.736792 mapread 0x1ea000 thru 0x1f22da (0x82db bytes) 000250[0] 1789608208.760026 read_OD 0x30e000 thru 0x319fff (0xc000 bytes) 000251[1] 1789608208.772796 mapwrite 0x36c000 thru 0x37022d (0x422e bytes) 000253[0] 1789608208.894829 mapwrite 0x4d7000 thru 0x4e1e53 (0xae54 bytes) 000254[1] 1789608209.013039 punch from 0x3f007b to 0x3ff1d3 (0xf159 bytes) 000255[0] 1789608209.030846 write_OD 0x07f000 thru 0x080fff (0x2000 bytes) 000256[0] 1789608209.079755 mapread 0x209000 thru 0x218832 (0xf833 bytes) 000257[1] 1789608209.119930 write_OD 0x494000 thru 0x499fff (0x6000 bytes) 000260[0] 1789608209.180185 trunc from 0x4e1e54 to 0x2cde0d (0x214046 bytes) 000262[0] 1789608209.215713 trunc from 0x2cde0e to 0x3869a3 (0xb8b96 bytes) 000264[1] 1789608209.274157 write_OD 0x153000 thru 0x15efff (0xc000 bytes) 000266[0] 1789608209.354389 mapwrite 0x915000 thru 0x91d701 (0x8702 bytes) 000267[0] 1789608209.603242 write_OD 0x601000 thru 0x608fff (0x8000 bytes) 000268[1] 1789608209.689383 mapwrite 0x80e000 thru 0x8149e2 (0x69e3 bytes) 000271[1] 1789608209.891608 read_OD 0x412000 thru 0x41afff (0x9000 bytes) 000272[1] 1789608209.925381 mapread 0x273000 thru 0x273c73 (0xc74 bytes) 000274[1] 1789608209.943269 trunc from 0x91d702 to 0x747061 (0x1d66a0 bytes) 000275[0] 1789608210.002486 punch from 0x314258 to 0x318708 (0x44b1 bytes) 000278[1] 1789608210.099252 mapwrite 0x683000 thru 0x685d1b (0x2d1c bytes) 000281[1] 1789608210.257228 trunc from 0x747062 to 0x3dfbbd (0x3674a4 bytes) 000283[0] 1789608210.302955 read_OD 0x023000 thru 0x023fff (0x1000 bytes) 000286[1] 1789608210.329148 mapwrite 0x3c3000 thru 0x3cd3d7 (0xa3d8 bytes) 000287[0] 1789608210.545993 mapread 0x161000 thru 0x1662b9 (0x52ba bytes) 000289[0] 1789608210.646406 read_OD 0x1a9000 thru 0x1acfff (0x4000 bytes) 000291[1] 1789608210.658417 read_OD 0x1eb000 thru 0x1ecfff (0x2000 bytes) 000295[1] 1789608210.668491 mapwrite 0x8ce000 thru 0x8d3243 (0x5244 bytes) 000296[1] 1789608211.102125 mapwrite 0x781000 thru 0x782162 (0x1163 bytes) 000299[0] 1789608211.250198 write_OD 0x66f000 thru 0x673fff (0x5000 bytes) 000300[0] 1789608211.398012 write_OD 0x0d6000 thru 0x0d8fff (0x3000 bytes) 000303[1] 1789608211.519836 write_OD 0x4a0000 thru 0x4a5fff (0x6000 bytes) 000307[1] 1789608211.626787 mapread 0x88c000 thru 0x89564e (0x964f bytes) 000308[0] 1789608211.759107 punch from 0x8cc987 to 0x8d1906 (0x4f80 bytes) 000312[0] 1789608211.847426 read_OD 0x2db000 thru 0x2dcfff (0x2000 bytes) 000313[0] 1789608211.868952 read_OD 0x119000 thru 0x120fff (0x8000 bytes) 000317[1] 1789608211.885278 write_OD 0x210000 thru 0x217fff (0x8000 bytes) 000318[1] 1789608211.994801 punch from 0x836d01 to 0x84189e (0xab9e bytes) 000321[1] 1789608212.087774 read_OD 0x67e000 thru 0x687fff (0xa000 bytes) 000322[1] 1789608212.109878 read_OD 0x0ef000 thru 0x0f8fff (0xa000 bytes) 000324[1] 1789608212.131355 trunc from 0x8d3244 to 0x8848a7 (0x4e99c bytes) 000325[0] 1789608212.205702 punch from 0x7ffa1c to 0x80009d (0x682 bytes) 000326[1] 1789608212.249028 trunc from 0x8848a8 to 0x78a26b (0xfa63c bytes) 000327[0] 1789608212.290163 trunc from 0x78a26c to 0x48421f (0x30604c bytes) 000328[1] 1789608212.344421 read_OD 0x28a000 thru 0x28cfff (0x3000 bytes) 000331[0] 1789608212.382396 trunc from 0x484220 to 0x0f6f45 (0x38d2da bytes) 000332[1] 1789608212.427371 write_OD 0x4c3000 thru 0x4ccfff (0xa000 bytes) 000333[0] 1789608212.494388 mapwrite 0x95d000 thru 0x960281 (0x3282 bytes) 000334[0] 1789608212.612661 trunc from 0x960282 to 0x41bba3 (0x5446de bytes) 000335[1] 1789608212.638279 write_OD 0x201000 thru 0x204fff (0x4000 bytes) 000337[1] 1789608212.701815 write_OD 0x323000 thru 0x32ffff (0xd000 bytes) 000339[1] 1789608212.766280 trunc from 0x41bba4 to 0x224ace (0x1f70d5 bytes) 000341[1] 1789608212.813637 mapwrite 0x4d6000 thru 0x4e3042 (0xd043 bytes) 000342[0] 1789608212.920683 trunc from 0x4e3043 to 0x4f8464 (0x15422 bytes) 000344[1] 1789608212.981366 read_OD 0x28f000 thru 0x29cfff (0xe000 bytes) 000346[0] 1789608213.017163 mapread 0x054000 thru 0x061135 (0xd136 bytes) 000350[0] 1789608213.048319 punch from 0x3baba5 to 0x3c4eaf (0xa30b bytes) 000351[1] 1789608213.083259 mapwrite 0x645000 thru 0x64ec2d (0x9c2e bytes) 000353[1] 1789608213.207387 trunc from 0x64ec2e to 0x1d9d6d (0x474ec0 bytes) 000354[1] 1789608213.247183 read_OD 0x05d000 thru 0x060fff (0x4000 bytes) 000355[0] 1789608213.266367 read_OD 0x077000 thru 0x07afff (0x4000 bytes) 000356[0] 1789608213.276012 read_OD 0x132000 thru 0x132fff (0x1000 bytes) 000357[1] 1789608213.280434 read_OD 0x041000 thru 0x04efff (0xe000 bytes) 000359[0] 1789608213.290029 read_OD 0x027000 thru 0x02bfff (0x5000 bytes) 000360[1] 1789608213.299433 trunc from 0x1d9d6e to 0x306cea (0x12cf7d bytes) 000363[1] 1789608213.339725 mapwrite 0x19f000 thru 0x1aabc7 (0xbbc8 bytes) 000364[0] 1789608213.444661 punch from 0x223bdb to 0x22bf54 (0x837a bytes) 000365[1] 1789608213.470451 punch from 0x0a8ead to 0x0b4ddc (0xbf30 bytes) 000367[1] 1789608213.505208 read_OD 0x29e000 thru 0x29efff (0x1000 bytes) 000370[1] 1789608213.524942 read_OD 0x278000 thru 0x284fff (0xd000 bytes) 000371[1] 1789608213.534394 write_OD 0x769000 thru 0x76afff (0x2000 bytes) 000372[1] 1789608213.588526 punch from 0x58ee9e to 0x598b55 (0x9cb8 bytes) 000373[1] 1789608213.604222 mapwrite 0x06a000 thru 0x073b04 (0x9b05 bytes) 000376[0] 1789608213.775669 read_OD 0x264000 thru 0x26ffff (0xc000 bytes) 000377[1] 1789608213.820020 punch from 0x5ace76 to 0x5ba374 (0xd4ff bytes) 000378[0] 1789608213.843533 punch from 0x4ba1a4 to 0x4bf050 (0x4ead bytes) 000381[0] 1789608213.866968 mapwrite 0x0d2000 thru 0x0db7ee (0x97ef bytes) 000386[0] 1789608213.996389 trunc from 0x76b000 to 0x47bcc9 (0x2ef336 bytes) 000387[0] 1789608214.075554 write_OD 0x3a8000 thru 0x3a8fff (0x1000 bytes) 000388[0] 1789608214.205886 write_OD 0x6da000 thru 0x6e3fff (0xa000 bytes) 000391[1] 1789608214.323181 trunc from 0x6e4000 to 0x0fb82d (0x5e87d2 bytes) 000392[1] 1789608214.405351 write_OD 0x8cd000 thru 0x8d7fff (0xb000 bytes) 000393[0] 1789608214.533422 trunc from 0x8d8000 to 0x60f194 (0x2c8e6b bytes) 000395[0] 1789608214.601452 mapread 0x5bb000 thru 0x5c2d15 (0x7d16 bytes) 000396[1] 1789608214.701871 mapread 0x13c000 thru 0x1416f4 (0x56f5 bytes) 000397[1] 1789608214.797663 read_OD 0x1fa000 thru 0x1fdfff (0x4000 bytes) 000398[1] 1789608214.808300 punch from 0x104ba8 to 0x1117e3 (0xcc3c bytes) 000399[1] 1789608214.835345 trunc from 0x60f195 to 0x0a1dad (0x56d3e7 bytes) 000400[1] 1789608214.854908 trunc from 0x0a1dae to 0x1eb44c (0x14969f bytes) 000403[1] 1789608214.875291 mapread 0x140000 thru 0x14aa0b (0xaa0c bytes) 000404[1] 1789608214.923981 punch from 0x0de2a5 to 0x0e044e (0x21aa bytes) 000407[1] 1789608214.931801 trunc from 0x1eb44d to 0x21a2e6 (0x2ee9a bytes) 000408[0] 1789608214.954855 mapwrite 0x41a000 thru 0x4204a0 (0x64a1 bytes) 000409[1] 1789608215.082250 mapread 0x102000 thru 0x10f983 (0xd984 bytes) 000410[1] 1789608215.133756 write_OD 0x5b5000 thru 0x5b9fff (0x5000 bytes) 000411[1] 1789608215.212213 mapread 0x2df000 thru 0x2e8893 (0x9894 bytes) 000413[1] 1789608215.240421 write_OD 0x0f2000 thru 0x0fdfff (0xc000 bytes) 000415[0] 1789608215.297562 mapwrite 0x7e2000 thru 0x7e8198 (0x6199 bytes) 000416[1] 1789608215.462362 punch from 0x7bc9b9 to 0x7bfdc9 (0x3411 bytes) 000417[1] 1789608215.485173 write_OD 0x377000 thru 0x377fff (0x1000 bytes) 000418[0] 1789608215.541806 write_OD 0x2fb000 thru 0x305fff (0xb000 bytes) 000422[1] 1789608215.618956 read_OD 0x5a1000 thru 0x5a6fff (0x6000 bytes) 000423[1] 1789608215.629449 trunc from 0x7e8199 to 0x1201e2 (0x6c7fb6 bytes) 000426[1] 1789608215.715054 punch from 0x081aac to 0x084b17 (0x306c bytes) 000430[1] 1789608215.754965 mapwrite 0x4d0000 thru 0x4d43d1 (0x43d2 bytes) 000432[0] 1789608215.924367 mapread 0x2c7000 thru 0x2d6d97 (0xfd98 bytes) 000433[0] 1789608216.005193 mapwrite 0x829000 thru 0x833fea (0xafeb bytes) 000436[1] 1789608216.221455 mapread 0x04b000 thru 0x053cc2 (0x8cc3 bytes) 000437[1] 1789608216.276776 punch from 0x33c8d9 to 0x3447fd (0x7f25 bytes) 000440[1] 1789608216.361741 mapread 0x354000 thru 0x35e8e2 (0xa8e3 bytes) 000441[0] 1789608216.431507 trunc from 0x833feb to 0x1839dc (0x6b060e bytes) 000442[1] 1789608216.550945 write_OD 0x516000 thru 0x51cfff (0x7000 bytes) 000444[1] 1789608216.745302 mapwrite 0x4d0000 thru 0x4d7c61 (0x7c62 bytes) 000446[0] 1789608217.057609 write_OD 0x3aa000 thru 0x3aafff (0x1000 bytes) 000447[0] 1789608217.257642 trunc from 0x51d000 to 0x525a95 (0x8a96 bytes) 000448[1] 1789608217.289190 read_OD 0x1ee000 thru 0x1f2fff (0x5000 bytes) 000449[1] 1789608217.322562 mapread 0x3f7000 thru 0x3f991e (0x291f bytes) 000450[0] 1789608217.350275 mapwrite 0x15e000 thru 0x163b01 (0x5b02 bytes) 000451[1] 1789608217.512775 trunc from 0x525a96 to 0x9d26a5 (0x4acc10 bytes) 000452[0] 1789608217.612872 read_OD 0x677000 thru 0x67afff (0x4000 bytes) 000453[0] 1789608217.652329 trunc from 0x9d26a6 to 0x3cb582 (0x607123 bytes) 000454[0] 1789608217.716713 mapwrite 0x1fd000 thru 0x206f7e (0x9f7f bytes) 000455[1] 1789608217.830161 mapwrite 0x366000 thru 0x3728df (0xc8e0 bytes) 000456[0] 1789608217.957020 mapread 0x128000 thru 0x130d77 (0x8d78 bytes) 000459[1] 1789608218.002924 trunc from 0x3cb583 to 0x6df77d (0x3141fb bytes) 000460[1] 1789608218.039215 mapwrite 0x7b6000 thru 0x7c41ad (0xe1ae bytes) 000461[1] 1789608218.164359 write_OD 0x623000 thru 0x629fff (0x7000 bytes) 000464[1] 1789608218.228991 trunc from 0x7c41ae to 0x4ede3c (0x2d6371 bytes) 000465[1] 1789608218.270412 trunc from 0x4ede3d to 0x5553ac (0x67570 bytes) 000468[1] 1789608218.306276 trunc from 0x5553ad to 0x9949d9 (0x43f62d bytes) 000469[0] 1789608218.347234 punch from 0x7dcdab to 0x7e73d6 (0xa62c bytes) 000470[0] 1789608218.387850 trunc from 0x9949da to 0x7c821c (0x1cc7bd bytes) 000471[1] 1789608218.414964 mapwrite 0x22d000 thru 0x235af1 (0x8af2 bytes) 000473[1] 1789608218.526414 punch from 0x1bd10a to 0x1bd37c (0x273 bytes) 000475[1] 1789608218.531175 mapread 0x211000 thru 0x218a6b (0x7a6c bytes) 000476[0] 1789608218.557150 punch from 0x17796f to 0x186388 (0xea1a bytes) 000477[0] 1789608218.573793 trunc from 0x7c821d to 0x33a83f (0x48d9dd bytes) 000479[0] 1789608218.640676 mapread 0x279000 thru 0x27aaac (0x1aad bytes) 000482[0] 1789608218.717663 mapwrite 0x136000 thru 0x13883c (0x283d bytes) 000483[0] 1789608218.838202 read_OD 0x123000 thru 0x12ffff (0xd000 bytes) 000484[1] 1789608218.869710 write_OD 0x0e9000 thru 0x0ebfff (0x3000 bytes) 000489[0] 1789608218.937271 mapread 0x045000 thru 0x046cfb (0x1cfc bytes) 000490[1] 1789608218.988727 mapread 0x1b7000 thru 0x1c53dc (0xe3dd bytes) 000491[1] 1789608219.020522 punch from 0x25a09f to 0x25b8be (0x1820 bytes) 000492[0] 1789608219.061402 mapread 0x183000 thru 0x18689b (0x389c bytes) 000494[0] 1789608219.107948 write_OD 0x202000 thru 0x206fff (0x5000 bytes) 000495[0] 1789608219.166672 read_OD 0x0eb000 thru 0x0ebfff (0x1000 bytes) 000498[0] 1789608219.179125 mapread 0x040000 thru 0x043dae (0x3daf bytes) 000499[1] 1789608219.237524 mapwrite 0x8a0000 thru 0x8a2462 (0x2463 bytes) All operations completed A-OK! PASS 16a (71s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 21:23:45 (1789608225) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.391436 s, 21.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.283572 s, 29.6 MB/s Chance of close/open is 1 in 50 Seed set to 8229 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1789608229.927691 write 0x2f220c thru 0x2f378c (0x1581 bytes) 000003[1] 1789608229.941293 mapread 0x1b073a thru 0x1b2497 (0x1d5e bytes) 000009[0] 1789608230.124429 punch from 0x1f260b to 0x1f3bce (0x15c4 bytes) 000010[1] 1789608230.176412 mapwrite 0x2b9f19 thru 0x2bb11b (0x1203 bytes) truncating to largest ever: 0x3133c7 000011[1] 1789608230.323317 trunc from 0x2f378d to 0x3133c6 (0x1fc3a bytes) 000013[0] 1789608230.385107 read 0x236e03 thru 0x2389dc (0x1bda bytes) 000016[1] 1789608230.454631 trunc from 0x3133c7 to 0x2907b1 (0x82c15 bytes) 000018[1] 1789608230.547916 read 0x261ca6 thru 0x261e81 (0x1dc bytes) 000020[0] 1789608230.556076 mapread 0x117ed4 thru 0x118270 (0x39d bytes) 000022[0] 1789608230.622930 punch from 0x041247 to 0x041985 (0x73f bytes) truncating to largest ever: 0x6f88d4 000023[0] 1789608230.652725 trunc from 0x2907b2 to 0x6f88d3 (0x468122 bytes) 000024[0] 1789608230.721063 write 0x619c50 thru 0x61ac05 (0xfb6 bytes) 000028[0] 1789608230.795552 mapread 0x430a9a thru 0x431dcd (0x1334 bytes) 000029[0] 1789608230.806899 trunc from 0x6f88d4 to 0x5d2aca (0x125e09 bytes) 000031[0] 1789608230.822932 mapwrite 0x419d86 thru 0x419d8a (0x5 bytes) 000032[0] 1789608230.983970 read 0x0567da thru 0x0576f3 (0xf1a bytes) 000034[0] 1789608231.036708 punch from 0x1cdfe4 to 0x1ced3e (0xd5b bytes) 000035[0] 1789608231.070047 mapwrite 0x7b44ce thru 0x7b4a4b (0x57e bytes) truncating to largest ever: 0x8cec70 000036[0] 1789608231.142552 trunc from 0x7b4a4c to 0x8cec6f (0x11a224 bytes) 000038[0] 1789608231.162738 mapread 0x117dcc thru 0x119506 (0x173b bytes) 000039[0] 1789608231.214082 mapread 0x634d62 thru 0x635bf6 (0xe95 bytes) 000040[0] 1789608231.226732 mapwrite 0x16df9d thru 0x16e7dc (0x840 bytes) 000041[1] 1789608231.283042 trunc from 0x8cec70 to 0x1e34b1 (0x6eb7be bytes) 000042[0] 1789608231.361326 read 0x0f6a4f thru 0x0f74c9 (0xa7b bytes) 000043[0] 1789608231.385802 mapread 0x117572 thru 0x119401 (0x1e90 bytes) truncating to largest ever: 0x91414f 000044[1] 1789608231.436532 trunc from 0x1e34b2 to 0x91414e (0x730c9d bytes) 000045[1] 1789608231.498146 mapread 0x3441d5 thru 0x344da4 (0xbd0 bytes) 000046[1] 1789608231.516035 mapread 0x5f841e thru 0x5f93f1 (0xfd4 bytes) 000047[1] 1789608231.528250 trunc from 0x91414f to 0x2524aa (0x6c1ca4 bytes) 000048[1] 1789608231.560517 mapwrite 0x8a27de thru 0x8a2bee (0x411 bytes) 000049[1] 1789608231.628717 read 0x705030 thru 0x705b3e (0xb0f bytes) 000051[1] 1789608231.633604 punch from 0x55698c to 0x558771 (0x1de6 bytes) 000052[0] 1789608231.640809 punch from 0x84c3a4 to 0x84caa1 (0x6fe bytes) 000053[0] 1789608231.676803 mapread 0x0912c2 thru 0x091fc0 (0xcff bytes) 000054[1] 1789608231.709484 write 0x4d9a55 thru 0x4daa77 (0x1023 bytes) 000055[1] 1789608231.740622 mapread 0x52af27 thru 0x52ca06 (0x1ae0 bytes) 000056[1] 1789608231.789867 write 0x7fddc7 thru 0x7fdeb2 (0xec bytes) 000057[1] 1789608231.817487 punch from 0x6ea817 to 0x6eba48 (0x1232 bytes) 000058[1] 1789608231.843672 mapread 0x3a5f26 thru 0x3a6f4c (0x1027 bytes) 000059[1] 1789608231.888874 trunc from 0x8a2bef to 0x7f3a32 (0xaf1bc bytes) 000061[1] 1789608232.031301 trunc from 0x7f3a33 to 0x55dc1e (0x295e14 bytes) 000062[0] 1789608232.055285 punch from 0x020ab8 to 0x0226dd (0x1c26 bytes) 000063[1] 1789608232.092710 punch from 0x1e10dc to 0x1e2cc1 (0x1be6 bytes) 000067[0] 1789608232.188845 write 0x351e1a thru 0x353db4 (0x1f9b bytes) 000068[1] 1789608232.236878 write 0x0c4355 thru 0x0c4935 (0x5e1 bytes) 000069[0] 1789608232.282198 punch from 0x3f7d64 to 0x3f8552 (0x7ef bytes) 000073[1] 1789608232.352423 mapwrite 0x0720fc thru 0x07272a (0x62f bytes) 000075[0] 1789608232.536228 read 0x3a8da7 thru 0x3a92d1 (0x52b bytes) 000076[1] 1789608232.565305 mapwrite 0x6eec29 thru 0x6f080d (0x1be5 bytes) 000077[0] 1789608232.685819 mapwrite 0x9f9d36 thru 0x9fbc61 (0x1f2c bytes) 000078[0] 1789608232.855492 trunc from 0x9fbc62 to 0x0b572e (0x946533 bytes) 000080[1] 1789608232.947096 read 0x060568 thru 0x062515 (0x1fae bytes) 000083[0] 1789608233.045019 punch from 0x047806 to 0x04864b (0xe46 bytes) 000086[1] 1789608233.143245 mapread 0x0a562b thru 0x0a6ee2 (0x18b8 bytes) 000087[1] 1789608233.217740 mapread 0x00f5a1 thru 0x0102d3 (0xd33 bytes) 000088[0] 1789608233.263109 write 0x8fee5d thru 0x8ffafb (0xc9f bytes) 000090[1] 1789608233.325131 mapread 0x654503 thru 0x6563da (0x1ed8 bytes) 000096[1] 1789608233.525747 mapwrite 0x1169b2 thru 0x118494 (0x1ae3 bytes) 000097[1] 1789608233.773446 trunc from 0x8ffafc to 0x23b977 (0x6c4184 bytes) 000098[0] 1789608233.879611 punch from 0x0e5d54 to 0x0e7b2a (0x1dd7 bytes) 000099[1] 1789608233.916159 write 0x4618b1 thru 0x4628aa (0xffa bytes) 000102[1] 1789608233.968463 write 0x87e3db thru 0x87ea10 (0x636 bytes) 000104[0] 1789608234.013131 mapwrite 0x65e93e thru 0x65ec21 (0x2e4 bytes) 000107[1] 1789608234.276494 trunc from 0x87ea11 to 0x2a56d6 (0x5d933a bytes) 000108[0] 1789608234.307683 write 0x056198 thru 0x056880 (0x6e9 bytes) 000110[0] 1789608234.363543 read 0x045e8d thru 0x047431 (0x15a5 bytes) 000111[1] 1789608234.450622 mapwrite 0x150278 thru 0x151b62 (0x18eb bytes) 000112[0] 1789608234.561704 mapread 0x1db851 thru 0x1dd71a (0x1eca bytes) 000113[0] 1789608234.617874 trunc from 0x2a56d7 to 0x74400f (0x49e939 bytes) 000114[0] 1789608234.705958 mapwrite 0x4b0f34 thru 0x4b282d (0x18fa bytes) 000115[1] 1789608234.856066 read 0x3bdf75 thru 0x3be62d (0x6b9 bytes) 000116[0] 1789608234.935511 mapread 0x4d1201 thru 0x4d1852 (0x652 bytes) 000117[0] 1789608234.958477 trunc from 0x744010 to 0x4e53d4 (0x25ec3b bytes) 000118[0] 1789608235.015239 punch from 0x3aee58 to 0x3afc74 (0xe1d bytes) 000121[0] 1789608235.068159 mapread 0x4a73c8 thru 0x4a7cdc (0x915 bytes) 000122[1] 1789608235.070066 punch from 0x4d7968 to 0x4d9778 (0x1e11 bytes) 000123[1] 1789608235.105705 mapwrite 0x9beff3 thru 0x9bfc28 (0xc36 bytes) 000125[0] 1789608235.257503 mapwrite 0x3f55c3 thru 0x3f6794 (0x11d2 bytes) 000127[1] 1789608235.403772 read 0x313562 thru 0x314c79 (0x1718 bytes) 000128[1] 1789608235.432289 punch from 0x46dc18 to 0x46f439 (0x1822 bytes) 000129[1] 1789608235.470273 write 0x9f6f69 thru 0x9f8462 (0x14fa bytes) 000130[0] 1789608235.524227 mapwrite 0x7305ef thru 0x73123e (0xc50 bytes) 000132[0] 1789608235.742808 punch from 0x797f4d to 0x799530 (0x15e4 bytes) 000133[0] 1789608235.771245 read 0x489208 thru 0x48a6cf (0x14c8 bytes) 000134[0] 1789608235.797480 write 0x86ee29 thru 0x86f77e (0x956 bytes) 000137[1] 1789608235.840832 read 0x10a9bd thru 0x10bd13 (0x1357 bytes) 000139[1] 1789608235.901411 trunc from 0x9f8463 to 0x1e9590 (0x80eed2 bytes) 000140[0] 1789608235.963071 mapread 0x039755 thru 0x03ae41 (0x16ed bytes) 000141[0] 1789608236.012797 write 0x5db7e9 thru 0x5dbc8d (0x4a5 bytes) 000144[1] 1789608236.089450 punch from 0x39b414 to 0x39b968 (0x555 bytes) 000145[1] 1789608236.127088 trunc from 0x5dbc8e to 0x20849a (0x3d37f3 bytes) 000146[1] 1789608236.165646 mapwrite 0x6e8d25 thru 0x6e914e (0x42a bytes) 000147[0] 1789608236.286735 mapwrite 0x5bc2ed thru 0x5be0f0 (0x1e04 bytes) truncating to largest ever: 0x9ff958 000148[0] 1789608236.434546 trunc from 0x6e914f to 0x9ff957 (0x316809 bytes) 000149[1] 1789608236.472492 punch from 0x955a8e to 0x956083 (0x5f6 bytes) 000150[1] 1789608236.522709 write 0x90613c thru 0x908116 (0x1fdb bytes) 000151[0] 1789608236.560318 write 0x788872 thru 0x7899f0 (0x117f bytes) 000152[1] 1789608236.599160 punch from 0x5ba9cf to 0x5bbac9 (0x10fb bytes) 000153[1] 1789608236.631206 trunc from 0x9ff958 to 0x497b07 (0x567e50 bytes) 000154[0] 1789608236.671681 mapread 0x2c766d thru 0x2c81c3 (0xb57 bytes) 000158[0] 1789608236.744137 write 0x9b22ef thru 0x9b3d81 (0x1a93 bytes) 000161[0] 1789608236.822468 mapread 0x5304cd thru 0x531de5 (0x1919 bytes) 000162[1] 1789608236.867038 trunc from 0x9b3d82 to 0x8fce5c (0xb6f25 bytes) 000165[0] 1789608236.961232 punch from 0x2af0cb to 0x2af35a (0x290 bytes) 000166[0] 1789608236.994673 punch from 0x82c313 to 0x82e2d1 (0x1fbf bytes) 000167[1] 1789608237.035895 trunc from 0x8fce5d to 0x418e99 (0x4e3fc3 bytes) 000168[1] 1789608237.102744 write 0x3c4e5b thru 0x3c58ff (0xaa5 bytes) 000169[1] 1789608237.143546 punch from 0x07d505 to 0x07e4ed (0xfe9 bytes) 000173[1] 1789608237.231236 trunc from 0x418e9a to 0x9c30b4 (0x5aa21b bytes) 000174[1] 1789608237.293467 mapread 0x3e7c64 thru 0x3e94b1 (0x184e bytes) 000175[0] 1789608237.326116 punch from 0x7d1be0 to 0x7d1f08 (0x329 bytes) 000176[1] 1789608237.360624 read 0x9bcf40 thru 0x9bec65 (0x1d26 bytes) 000178[1] 1789608237.406429 trunc from 0x9c30b5 to 0x94d03c (0x76078 bytes) 000180[1] 1789608237.474974 punch from 0x6d051d to 0x6d0ccf (0x7b3 bytes) 000182[0] 1789608237.515956 read 0x280ba9 thru 0x281a91 (0xee9 bytes) 000183[0] 1789608237.574879 mapread 0x483b48 thru 0x484214 (0x6cd bytes) 000184[1] 1789608237.629416 trunc from 0x94d03d to 0x17502b (0x7d8011 bytes) 000185[0] 1789608237.671267 mapwrite 0x70d7aa thru 0x70e1e6 (0xa3d bytes) 000186[1] 1789608237.771780 write 0x97ba22 thru 0x97bfac (0x58b bytes) 000188[0] 1789608237.821075 trunc from 0x97bfad to 0x4b1463 (0x4cab49 bytes) 000189[1] 1789608237.882745 write 0x459bfb thru 0x459f79 (0x37f bytes) 000191[1] 1789608237.937280 trunc from 0x4b1464 to 0x78b8e5 (0x2da482 bytes) 000192[0] 1789608237.991492 trunc from 0x78b8e6 to 0x8b727e (0x12b999 bytes) 000193[0] 1789608238.092051 mapread 0x500ca9 thru 0x501c4c (0xfa4 bytes) 000194[1] 1789608238.134529 mapread 0x112a28 thru 0x112d9e (0x377 bytes) 000202[1] 1789608238.360600 mapwrite 0x55968f thru 0x55a144 (0xab6 bytes) 000204[0] 1789608238.526905 read 0x2535f8 thru 0x254cc2 (0x16cb bytes) 000205[0] 1789608238.566754 punch from 0x173a59 to 0x173f8d (0x535 bytes) 000207[1] 1789608238.655504 mapread 0x0911f8 thru 0x092408 (0x1211 bytes) 000208[0] 1789608238.715542 write 0x433436 thru 0x433f1c (0xae7 bytes) 000209[0] 1789608238.767156 read 0x52541f thru 0x5255a2 (0x184 bytes) 000210[1] 1789608238.782260 write 0x7e7a1f thru 0x7e8a70 (0x1052 bytes) 000212[1] 1789608238.842632 punch from 0x8b216f to 0x8b3cdc (0x1b6e bytes) 000213[1] 1789608238.914250 mapwrite 0x4d3e05 thru 0x4d5c24 (0x1e20 bytes) 000214[0] 1789608239.017152 write 0x12ca87 thru 0x12d274 (0x7ee bytes) 000215[1] 1789608239.060696 trunc from 0x8b727f to 0x0dfb39 (0x7d7745 bytes) 000216[1] 1789608239.100484 mapwrite 0x8f81fb thru 0x8f97f9 (0x15ff bytes) 000218[1] 1789608239.273003 trunc from 0x8f97fa to 0x78ddab (0x16ba4e bytes) 000219[0] 1789608239.321008 write 0x04742d thru 0x04921f (0x1df3 bytes) 000220[0] 1789608239.362195 mapwrite 0x471b78 thru 0x472e9f (0x1328 bytes) 000221[1] 1789608239.464000 write 0x61270e thru 0x61359e (0xe91 bytes) 000222[1] 1789608239.514321 trunc from 0x78ddac to 0x0e2221 (0x6abb8a bytes) 000223[0] 1789608239.565107 trunc from 0x0e2222 to 0x7efe43 (0x70dc22 bytes) 000225[0] 1789608239.711994 mapread 0x1ea7cf thru 0x1eaedf (0x711 bytes) 000228[1] 1789608239.793124 mapwrite 0x32f4e8 thru 0x330a36 (0x154f bytes) 000230[1] 1789608239.994518 write 0x940182 thru 0x941332 (0x11b1 bytes) 000233[0] 1789608240.063082 mapwrite 0x2a8e58 thru 0x2a9173 (0x31c bytes) 000234[1] 1789608240.248280 punch from 0x32399c to 0x325008 (0x166d bytes) 000237[0] 1789608240.341087 read 0x2338a9 thru 0x2350bf (0x1817 bytes) 000241[0] 1789608240.391702 write 0x7b054d thru 0x7b121c (0xcd0 bytes) 000242[0] 1789608240.404676 mapwrite 0x3e47a9 thru 0x3e56c9 (0xf21 bytes) 000245[1] 1789608240.567149 read 0x2d0ba4 thru 0x2d193f (0xd9c bytes) 000246[0] 1789608240.592098 trunc from 0x941333 to 0x6f1d66 (0x24f5cc bytes) 000247[0] 1789608240.649496 mapread 0x212da7 thru 0x214ca2 (0x1efc bytes) 000248[0] 1789608240.691718 punch from 0x46b67d to 0x46c5a1 (0xf25 bytes) 000249[1] 1789608240.726900 trunc from 0x6f1d67 to 0x2aa4cf (0x447897 bytes) 000251[0] 1789608240.794524 mapwrite 0x09ed10 thru 0x0a0757 (0x1a48 bytes) 000253[1] 1789608240.905100 trunc from 0x2aa4d0 to 0x280347 (0x2a188 bytes) 000257[1] 1789608241.049340 mapread 0x08063a thru 0x081062 (0xa29 bytes) 000259[0] 1789608241.180352 punch from 0x132e79 to 0x133dbb (0xf43 bytes) 000260[0] 1789608241.212444 mapread 0x07ff02 thru 0x081501 (0x1600 bytes) 000262[1] 1789608241.274147 trunc from 0x280348 to 0x279732 (0x6c15 bytes) 000265[1] 1789608241.392732 read 0x10db81 thru 0x10e2c1 (0x741 bytes) 000267[0] 1789608241.493153 read 0x21a933 thru 0x21c5f9 (0x1cc7 bytes) 000268[1] 1789608241.580106 read 0x26574c thru 0x266666 (0xf1b bytes) 000269[1] 1789608241.664980 mapwrite 0x9c201c thru 0x9c33d7 (0x13bc bytes) 000270[0] 1789608241.957464 punch from 0x04fc9a to 0x050b7f (0xee6 bytes) 000271[0] 1789608242.006301 read 0x57002a thru 0x57017d (0x154 bytes) 000274[1] 1789608242.107086 write 0x6fa23d thru 0x6fb3a4 (0x1168 bytes) 000275[1] 1789608242.202442 mapwrite 0x26e1f9 thru 0x26f38d (0x1195 bytes) 000277[1] 1789608242.358116 mapread 0x449c79 thru 0x44ab53 (0xedb bytes) 000278[0] 1789608242.421524 mapread 0x06ff14 thru 0x071437 (0x1524 bytes) 000279[0] 1789608242.498186 punch from 0x193af7 to 0x19589f (0x1da9 bytes) 000280[1] 1789608242.538828 read 0x1f7d00 thru 0x1f9831 (0x1b32 bytes) 000282[1] 1789608242.579037 mapread 0x536b0c thru 0x536cfb (0x1f0 bytes) 000283[1] 1789608242.600202 read 0x17cb85 thru 0x17e4e8 (0x1964 bytes) 000284[0] 1789608242.608915 trunc from 0x9c33d8 to 0x4c7e9c (0x4fb53b bytes) 000286[0] 1789608242.723100 write 0x0fdd36 thru 0x0feed2 (0x119d bytes) 000289[0] 1789608242.736595 mapwrite 0x99f2e6 thru 0x99fd41 (0xa5c bytes) 000293[1] 1789608242.855005 trunc from 0x99fd42 to 0x825387 (0x17a9ba bytes) 000302[0] 1789608243.020078 read 0x4f9037 thru 0x4f9dbe (0xd88 bytes) 000303[1] 1789608243.039381 mapread 0x3d613a thru 0x3d7a2e (0x18f5 bytes) 000305[1] 1789608243.051988 trunc from 0x825388 to 0x1057ba (0x71fbcd bytes) 000307[0] 1789608243.129743 write 0x76c9c9 thru 0x76dd9d (0x13d5 bytes) 000308[0] 1789608243.168222 trunc from 0x76dd9e to 0x36d03f (0x400d5e bytes) 000310[0] 1789608243.284678 punch from 0x0dd378 to 0x0dea2d (0x16b6 bytes) 000312[0] 1789608243.329539 trunc from 0x36d040 to 0x8fde5d (0x590e1e bytes) 000315[1] 1789608243.415437 write 0x8112c7 thru 0x81307b (0x1db5 bytes) 000316[1] 1789608243.459032 mapwrite 0x13ba4b thru 0x13bb89 (0x13f bytes) 000317[1] 1789608243.550248 read 0x60ed94 thru 0x61029f (0x150c bytes) 000321[0] 1789608243.616199 write 0x2b8397 thru 0x2b9209 (0xe73 bytes) 000322[1] 1789608243.646151 trunc from 0x8fde5e to 0x57217d (0x38bce0 bytes) 000324[0] 1789608243.763934 punch from 0x0c6d9d to 0x0c7c5e (0xec2 bytes) 000325[1] 1789608243.808042 mapwrite 0x93e028 thru 0x93f5df (0x15b8 bytes) 000326[0] 1789608244.004647 read 0x0de1b9 thru 0x0de59e (0x3e6 bytes) 000327[0] 1789608244.039022 trunc from 0x93f5e0 to 0x3b4ea4 (0x58a73b bytes) 000328[1] 1789608244.099518 write 0x5a967c thru 0x5ab3dd (0x1d62 bytes) 000329[1] 1789608244.149873 read 0x480a6c thru 0x481369 (0x8fe bytes) 000332[0] 1789608244.226456 write 0x7b1824 thru 0x7b370d (0x1eea bytes) 000334[0] 1789608244.294733 read 0x27528b thru 0x276589 (0x12ff bytes) 000336[0] 1789608244.329161 mapwrite 0x6ff447 thru 0x70107a (0x1c34 bytes) 000338[0] 1789608244.446451 mapread 0x5f8c7b thru 0x5fa216 (0x159c bytes) 000339[1] 1789608244.521327 mapread 0x1e27fa thru 0x1e397e (0x1185 bytes) 000340[0] 1789608244.556903 read 0x6a3647 thru 0x6a4808 (0x11c2 bytes) 000341[1] 1789608244.577704 read 0x454f5a thru 0x455c8c (0xd33 bytes) 000344[1] 1789608244.599472 write 0x38ef93 thru 0x390bb4 (0x1c22 bytes) 000346[0] 1789608244.633982 trunc from 0x7b370e to 0x58cd9f (0x22696e bytes) 000347[1] 1789608244.736666 trunc from 0x58cda0 to 0x2a4c9a (0x2e8105 bytes) 000348[0] 1789608244.813660 read 0x136ada thru 0x137806 (0xd2d bytes) 000349[1] 1789608244.844654 trunc from 0x2a4c9b to 0x1b93b4 (0xeb8e6 bytes) 000350[0] 1789608244.895607 write 0x819a5e thru 0x81b75c (0x1cff bytes) 000352[1] 1789608244.966451 write 0x9bfff2 thru 0x9c05e4 (0x5f3 bytes) 000353[1] 1789608244.994984 mapread 0x1037f8 thru 0x104998 (0x11a1 bytes) 000354[1] 1789608245.015464 mapwrite 0x7424d1 thru 0x743d26 (0x1856 bytes) 000355[1] 1789608245.210254 trunc from 0x9c05e5 to 0x43d019 (0x5835cb bytes) 000356[1] 1789608245.267283 mapwrite 0x4bfcac thru 0x4c08de (0xc33 bytes) 000357[0] 1789608245.428315 punch from 0x0686e9 to 0x069ccb (0x15e3 bytes) 000361[1] 1789608245.652145 mapwrite 0x4004fe thru 0x400d29 (0x82c bytes) 000362[1] 1789608245.742435 mapwrite 0x64bb78 thru 0x64d212 (0x169b bytes) 000363[0] 1789608245.959390 mapwrite 0x2582fc thru 0x259430 (0x1135 bytes) 000364[1] 1789608246.141424 trunc from 0x64d213 to 0x54c653 (0x100bbf bytes) 000369[1] 1789608246.240823 trunc from 0x54c654 to 0x605ec3 (0xb9870 bytes) 000371[1] 1789608246.348501 mapread 0x237340 thru 0x2378a1 (0x562 bytes) 000373[1] 1789608246.358372 write 0x7b1e50 thru 0x7b3865 (0x1a16 bytes) 000375[1] 1789608246.384790 punch from 0x1b400d to 0x1b43d6 (0x3ca bytes) 000377[0] 1789608246.429837 trunc from 0x7b3866 to 0x202f9d (0x5b08c8 bytes) 000378[0] 1789608246.508493 trunc from 0x202f9e to 0x666e60 (0x463ec3 bytes) 000379[1] 1789608246.550466 read 0x4597af thru 0x45a0e3 (0x935 bytes) 000380[1] 1789608246.636033 mapwrite 0x0e4169 thru 0x0e5c9f (0x1b37 bytes) 000384[1] 1789608246.874673 mapread 0x5b2d07 thru 0x5b469b (0x1995 bytes) 000386[1] 1789608246.939356 read 0x42b773 thru 0x42cd68 (0x15f6 bytes) 000387[1] 1789608246.952515 mapwrite 0x198043 thru 0x199868 (0x1826 bytes) 000388[0] 1789608247.029337 read 0x5e904a thru 0x5eae37 (0x1dee bytes) 000389[0] 1789608247.035502 punch from 0x1b5e28 to 0x1b6601 (0x7da bytes) 000390[1] 1789608247.052175 read 0x4e61ca thru 0x4e66fc (0x533 bytes) 000393[0] 1789608247.082447 read 0x30eba1 thru 0x30ed1f (0x17f bytes) 000394[0] 1789608247.089143 mapwrite 0x7b4174 thru 0x7b50ad (0xf3a bytes) 000395[1] 1789608247.261199 mapread 0x417325 thru 0x41832b (0x1007 bytes) 000396[0] 1789608247.320009 mapwrite 0x6496b6 thru 0x64a294 (0xbdf bytes) 000397[1] 1789608247.406832 read 0x521b15 thru 0x52394d (0x1e39 bytes) 000403[1] 1789608247.500339 punch from 0x6c1a7c to 0x6c2edf (0x1464 bytes) 000404[1] 1789608247.559545 punch from 0x07d287 to 0x07e3cb (0x1145 bytes) 000407[0] 1789608247.621021 read 0x17d598 thru 0x17edab (0x1814 bytes) 000410[0] 1789608247.668068 mapread 0x22419b thru 0x2258e4 (0x174a bytes) 000411[0] 1789608247.698702 read 0x10d24a thru 0x10e027 (0xdde bytes) 000413[1] 1789608247.721342 read 0x6614c6 thru 0x662f12 (0x1a4d bytes) 000415[0] 1789608247.746250 trunc from 0x7b50ae to 0x357dd1 (0x45d2dc bytes) 000417[0] 1789608247.887277 punch from 0x258db5 to 0x259d45 (0xf91 bytes) 000418[0] 1789608247.896107 mapread 0x0a78c8 thru 0x0a8af3 (0x122c bytes) 000420[0] 1789608247.977434 trunc from 0x357dd2 to 0x2031ee (0x154be3 bytes) 000421[0] 1789608248.007501 mapread 0x17550d thru 0x176a27 (0x151b bytes) 000422[1] 1789608248.019384 mapwrite 0x5c7e4c thru 0x5c99cf (0x1b84 bytes) 000424[1] 1789608248.284412 read 0x40e75e thru 0x40fac2 (0x1365 bytes) 000425[1] 1789608248.296998 mapwrite 0x28a5b6 thru 0x28b428 (0xe73 bytes) 000427[1] 1789608248.418863 mapread 0x51c18d thru 0x51cb96 (0xa0a bytes) 000428[1] 1789608248.435857 punch from 0x1653b3 to 0x16729d (0x1eeb bytes) 000430[1] 1789608248.472138 read 0x40c4c4 thru 0x40c7b4 (0x2f1 bytes) 000431[0] 1789608248.478664 punch from 0x30734a to 0x30918a (0x1e41 bytes) 000435[1] 1789608248.559422 punch from 0x0dcff1 to 0x0dec76 (0x1c86 bytes) 000437[1] 1789608248.605510 trunc from 0x5c99d0 to 0x3bde00 (0x20bbcf bytes) 000440[1] 1789608248.689236 mapread 0x37690c thru 0x376996 (0x8b bytes) 000441[0] 1789608248.708507 read 0x0c68b9 thru 0x0c7062 (0x7aa bytes) 000443[1] 1789608248.845451 mapread 0x223bd5 thru 0x224ef2 (0x131e bytes) 000445[1] 1789608248.897687 read 0x1e7902 thru 0x1e9791 (0x1e90 bytes) 000450[0] 1789608249.132748 read 0x122827 thru 0x122acb (0x2a5 bytes) 000452[1] 1789608249.176245 write 0x2f116e thru 0x2f1bd6 (0xa69 bytes) 000454[0] 1789608249.234912 punch from 0x11f4ca to 0x120b82 (0x16b9 bytes) 000455[1] 1789608249.303315 mapread 0x0d37cf thru 0x0d3fc7 (0x7f9 bytes) 000457[0] 1789608249.350461 punch from 0x2a201d to 0x2a2e99 (0xe7d bytes) 000458[1] 1789608249.424061 punch from 0x27f9cd to 0x280077 (0x6ab bytes) 000459[1] 1789608249.503556 read 0x1d4694 thru 0x1d530d (0xc7a bytes) 000462[0] 1789608249.632694 write 0x7ac24b thru 0x7acecc (0xc82 bytes) 000463[0] 1789608249.680012 trunc from 0x7acecd to 0x8de69b (0x1317cf bytes) 000464[1] 1789608249.751707 write 0x34f216 thru 0x34f6b3 (0x49e bytes) 000466[1] 1789608249.831301 punch from 0x48696a to 0x486acd (0x164 bytes) 000467[0] 1789608249.853362 mapwrite 0x55b139 thru 0x55b50e (0x3d6 bytes) 000468[0] 1789608249.929286 mapwrite 0x2ef51f thru 0x2ef715 (0x1f7 bytes) 000469[1] 1789608250.096858 write 0x04c023 thru 0x04de76 (0x1e54 bytes) 000470[0] 1789608250.157624 punch from 0x2b939e to 0x2baa79 (0x16dc bytes) 000471[0] 1789608250.202434 mapwrite 0x42403c thru 0x425424 (0x13e9 bytes) 000475[0] 1789608250.420297 mapwrite 0x067dc6 thru 0x068de8 (0x1023 bytes) 000478[1] 1789608250.485669 trunc from 0x8de69c to 0x892f73 (0x4b728 bytes) 000479[1] 1789608250.588409 trunc from 0x892f74 to 0x3a1b1b (0x4f1458 bytes) 000483[0] 1789608250.624815 mapread 0x20e404 thru 0x20efc4 (0xbc1 bytes) 000484[0] 1789608250.730467 mapwrite 0x45f3df thru 0x45ffef (0xc11 bytes) 000486[0] 1789608250.924277 mapread 0x3cdaac thru 0x3cfa99 (0x1fee bytes) 000487[0] 1789608250.968384 punch from 0x112b07 to 0x11377c (0xc76 bytes) 000491[1] 1789608251.041987 mapwrite 0x5e7f52 thru 0x5e8ff6 (0x10a5 bytes) 000493[1] 1789608251.409893 write 0x37ceb9 thru 0x37e839 (0x1981 bytes) 000494[0] 1789608251.486844 mapwrite 0x060f44 thru 0x061e22 (0xedf bytes) 000496[0] 1789608251.634223 trunc from 0x5e8ff7 to 0x45266d (0x196989 bytes) 000498[1] 1789608251.700304 punch from 0x298567 to 0x29a286 (0x1d20 bytes) 000500[0] 1789608251.790509 write 0x99aa38 thru 0x99b934 (0xefd bytes) All operations completed A-OK! PASS 16b (32s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 21:24:18 (1789608258) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 21:24:23 (1789608263) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.494407 s, 17.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.495124 s, 16.9 MB/s ldlm.namespaces.MGC192.168.202.116@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9fe893100000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9fe8a00ea000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9fe893100000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff9fe8a00ea000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9fe893100000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9fe893100000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.9244 s, 8.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.95442 s, 26.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.15688 s, 33.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.47153 s, 12.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1311 s, 9.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93038 s, 21.3 MB/s PASS 16d (61s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 21:25:25 (1789608325) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.25379 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871249 s, 12.0 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 21:25:32 (1789608332) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 21:26:01 (1789608361) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 21:26:30 (1789608390) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 21:26:38 (1789608398) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 21:26:47 (1789608407) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.49941 s, 17.5 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57478 s, 16.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.13954 s, 8.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.75474 s, 14.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.34097 s, 11.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.01148 s, 13.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.87146 s, 9.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.01054 s, 13.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.0705 s, 12.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.44129 s, 7.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.55106 s, 7.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.63231 s, 10.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.93359 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.41789 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.46055 s, 7.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.89651 s, 9.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.12405 s, 8.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.35651 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.68709 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (42s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 21:27:28 (1789608448) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Wed Sep 16 21:28:20 EDT 2026 + fsx 2 done: rc=0 Wed Sep 16 21:28:21 EDT 2026 + fsx 3 done: rc=0 Wed Sep 16 21:28:21 EDT 2026 + fsx 4 done: rc=0 Wed Sep 16 21:28:21 EDT 2026 + fsx 5 done: rc=0 Wed Sep 16 21:28:21 EDT 2026 PASS 16k (61s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 21:28:29 (1789608509) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 21:28:41 (1789608521) mmap test1: basic mmap operation (PASS, 0.15554s) mmap test2: MAP_PRIVATE not write back (PASS, 0.10602s) mmap test3: concurrent mmap ops on two nodes (PASS, 7.256s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6002s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.13951s) mmap test6: check mmap write/read content on two nodes (PASS, 0.2112s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.277s) mmap test11: multi-thread shared mmap (PASS, 1.4952s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1269s) PASS 18 (25s) == sanityn test 19: test concurrent uncached read races ========================================================================= 21:29:06 (1789608546) oleg216-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 21:29:11 (1789608551) striped dir -i0 -c2 -H crush2 /mnt/lustre/d20.sanityn PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 21:29:19 (1789608559) striped dir -i1 -c2 -H all_char /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 21:29:26 (1789608566) now is 1789608569 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 new atime is 1789608630 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 21:30:37 (1789608637) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 519.5K 403 519.1K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 519.0K 337 518.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 114.5K 465 114.1K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 115.3K 468 114.9K 1% /mnt/lustre[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 519.5K 403 519.1K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 519.0K 337 518.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 114.5K 465 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 468 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.4M 2.1G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 7.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 531980 403 531577 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 531436 337 531099 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 117297 465 116832 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118100 468 117632 1% /mnt/lustre2[OST:1] filesystem_summary: 235204 740 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 519.5K 403 519.1K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 519.0K 337 518.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 114.5K 465 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 468 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 531980 403 531577 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 531436 337 531099 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 117297 465 116832 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118100 468 117632 1% /mnt/lustre[OST:1] filesystem_summary: 235204 740 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 531980 403 531577 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 531436 337 531099 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 117297 465 116832 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118100 468 117632 1% /mnt/lustre2[OST:1] filesystem_summary: 235204 740 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 21:30:45 (1789608645) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 21:30:52 (1789608652) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (9s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 21:31:01 (1789608661) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (9s) == sanityn test 26a: allow mtime to get older ============ 21:31:10 (1789608670) utime: good mknod times 1789608671 <= 1789608672 <= 1789608672 for /mnt/lustre/f26a utime: good mknod times 1789608671 <= 1789608672 <= 1789608672 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (9s) == sanityn test 26b: sync mtime between ost and mds ====== 21:31:19 (1789608679) PASS 26b (11s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 21:31:30 (1789608690) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279602 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-16 21:31:32.000000000 -0400 Birth: 2026-09-16 21:31:33.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279602 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-16 21:31:32.000000000 -0400 Birth: 2026-09-16 21:31:33.000000000 -0400 PASS 26c (11s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 21:31:43 (1789608703) striped dir -i0 -c2 -H crush /mnt/lustre/d30.sanityn 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 21:31:53 (1789608713) striped dir -i1 -c2 -H crush /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (10s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 21:32:03 (1789608723) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg216-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete striped dir -i1 -c2 -H all_char /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.116@tcp:/lustre 7542784 41984 7494656 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 21:32:28 (1789608748) -rw-r--r-- 1 root root 0 Sep 16 21:32 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 21:32:39 (1789608759) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 45206 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 21:32:48 (1789608768) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 21:32:56 (1789608776) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 21:33:00 (1789608780) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.55977 s, 33.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.18111 s, 44.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.20425 s, 23.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.53436 s, 20.7 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.47141 s, 21.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.6025 s, 20.1 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35216 s, 38.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41457 s, 37.1 MB/s PASS 32c (28s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 21:33:30 (1789608810) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 21:33:34 (1789608814) Stopping /mnt/lustre-mds1 (opts:) on oleg216-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (52s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 21:34:26 (1789608866) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 1 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (73s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 21:35:39 (1789608939) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 1.95 seconds: 51.16 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 1.80 seconds: 55.53 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1789608948 ; total 0 ; last 0) total: 100 unlinks in 7 seconds: 14.285714 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1789608955 ; total 0 ; last 0) total: 100 unlinks in 3 seconds: 33.333332 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (27s) == sanityn test 34: no lock timeout under IO ============= 21:36:07 (1789608967) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fe893100000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe893100000.ost_server_uuid in FULL state after 0 sec oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fe893100000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe893100000.ost_server_uuid in IDLE state after 0 sec oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fe893100000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe893100000.ost_server_uuid in FULL state after 0 sec oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fe893100000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe893100000.ost_server_uuid in IDLE state after 0 sec oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fe893100000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe893100000.ost_server_uuid in FULL state after 0 sec oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fe893100000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fe893100000.ost_server_uuid in IDLE state after 0 sec PASS 34 (150s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 21:38:36 (1789609116) striped dir -i1 -c2 -H crush /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 57239 57284 for 60 sec... - open/close 580 (time 1789609134.14 total 10.02 last 57.89) - open/close 1182 (time 1789609144.16 total 20.04 last 60.08) - open/close 1898 (time 1789609154.17 total 30.05 last 71.54) - open/close 2620 (time 1789609164.17 total 40.05 last 72.19) - open/close 3326 (time 1789609174.17 total 50.05 last 70.57) - open/close 3989 (time 1789609184.17 total 60.05 last 66.29) total: 4000 open/close in 60.18 seconds: 66.46 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.116@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.202.116@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (75s) == sanityn test 36: handle ESTALE/open-unlink correctly == 21:39:51 (1789609191) striped dir -i0 -c2 -H crush /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg216-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.15568 s, 24.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33655 s, 39.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7490560) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.00996 s, 26.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7490560) after_dd(7443456) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.54904 s, 33.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7439360) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50881 s, 34.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.57974 s, 33.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3306 s, 39.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(6) *** before(7492608) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.09959 s, 47.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.22106 s, 42.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(8) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.21488 s, 23.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.70288 s, 30.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7439360) after(7490560) PASS 36 (330s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 21:45:21 (1789609521) striped dir -i1 -c2 -H all_char /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 - create 1066 (time 1789609537.52 total 10.00 last 106.58) - create 2350 (time 1789609547.52 total 20.00 last 128.38) - create 3622 (time 1789609557.53 total 30.01 last 127.07) - create 5039 (time 1789609567.54 total 40.02 last 141.61) - create 6464 (time 1789609577.54 total 50.02 last 142.43) - create 7882 (time 1789609587.55 total 60.03 last 141.74) - create 9099 (time 1789609597.55 total 70.03 last 121.67) total: 10000 create in 76.09 seconds: 131.43 ops/second PASS 37 (90s) == sanityn test 39a: file mtime does not change after rename ========================================================== 21:46:52 (1789609612) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 21:46:59 (1789609619) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 21:47:07 (1789609627) repeat after cancel_lru_locks PASS 39c (11s) == sanityn test 39d: sync write should update mtime ====== 21:47:18 (1789609638) fail_loc=0x411 fail_loc=0 PASS 39d (11s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 21:47:32 (1789609652) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (21s) == sanityn test 40c: pdirops: link and others ======================================================================== 21:47:53 (1789609673) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (20s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 21:48:14 (1789609694) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (20s) == sanityn test 40e: pdirops: rename and others ======================================================================== 21:48:34 (1789609714) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 21:48:52 (1789609732) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 21:49:05 (1789609745) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 41b (18s) == sanityn test 41c: pdirops: create vs link ======================================================================== 21:49:23 (1789609763) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (17s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 21:49:40 (1789609780) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 21:49:55 (1789609795) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41e (17s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 21:50:12 (1789609812) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41f (16s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 21:50:29 (1789609829) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41g (16s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 21:50:45 (1789609845) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (15s) == sanityn test 41i: reint_open: create vs create ======== 21:51:00 (1789609860) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1080s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 22:09:00 (1789610940) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 22:09:14 (1789610954) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 22:09:29 (1789610969) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 22:09:44 (1789610984) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 22:09:57 (1789610997) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 22:10:11 (1789611011) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42f (16s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 22:10:27 (1789611027) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42g (16s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 22:10:43 (1789611043) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42h (16s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 22:10:59 (1789611059) PASS 43a (215s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 22:14:35 (1789611275) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43b (15s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 22:14:50 (1789611290) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43c (15s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 22:15:05 (1789611305) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 22:15:19 (1789611319) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 22:15:33 (1789611333) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 22:15:46 (1789611346) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (16s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 22:16:02 (1789611362) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43h (19s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 22:16:21 (1789611381) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43i (19s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 22:16:41 (1789611401) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (131s) == sanityn test 43k: unlink vs create ==================== 22:18:52 (1789611532) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1319s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 22:40:52 (1789612852) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Sep 16 22:41:00 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 22:41:05 (1789612865) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 22:41:19 (1789612879) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 22:41:33 (1789612893) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 22:41:47 (1789612907) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 22:42:02 (1789612922) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 22:42:16 (1789612936) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 22:42:30 (1789612950) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44h (16s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 22:42:46 (1789612966) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (15s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 22:43:02 (1789612982) PASS 45a (255s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 22:47:16 (1789613236) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45b (16s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 22:47:33 (1789613253) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45c (16s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 22:47:49 (1789613269) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 22:48:04 (1789613284) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 22:48:20 (1789613300) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (18s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 22:48:37 (1789613317) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (16s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 22:48:54 (1789613334) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (18s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 22:49:12 (1789613352) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45i (17s) == sanityn test 45j: read vs rename ====================== 22:49:29 (1789613369) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1475s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 23:14:04 (1789614844) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (15s) == sanityn test 46b: pdirops: link vs create ======================================================================== 23:14:19 (1789614859) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 46b (14s) == sanityn test 46c: pdirops: link vs link ======================================================================== 23:14:33 (1789614873) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (17s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 23:14:50 (1789614890) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46d (20s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 23:15:10 (1789614910) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 23:15:25 (1789614925) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 23:15:39 (1789614939) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 23:15:53 (1789614953) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46h (15s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 23:16:08 (1789614968) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (16s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 23:16:24 (1789614984) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (16s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 23:16:40 (1789615000) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 47b (16s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 23:16:56 (1789615016) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (14s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 23:17:10 (1789615030) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47d (15s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 23:17:25 (1789615045) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (15s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 23:17:40 (1789615060) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47f (14s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 23:17:54 (1789615074) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (20s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 23:18:14 (1789615094) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.140954 s, 72.6 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (15s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 23:18:29 (1789615109) 0+1 records in 0+1 records out 158 bytes copied, 0.045641 s, 3.5 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (236936) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 23:18:39 (1789615119) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0193754 s, 52.9 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118009 s, 8.9 MB/s 3145728 PASS 51b (22s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 23:19:01 (1789615141) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0176963 s, 57.9 kB/s PASS 51c (14s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 23:19:15 (1789615155) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609399 s, 17.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 23:19:24 (1789615164) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1324 0x52c 0x280000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1324 0x52c 0x280000401 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 23:19:34 (1789615174) llite.lustre-ffff9fe893100000.xattr_cache=1 llite.lustre-ffff9fe8a00ea000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762662 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 23:19:36.000000000 -0400 Modify: 2026-09-16 23:19:36.000000000 -0400 Change: 2026-09-16 23:19:36.000000000 -0400 Birth: 2026-09-16 23:19:36.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281284 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 23:19:43.000000000 -0400 Modify: 2026-09-16 23:19:45.000000000 -0400 Change: 2026-09-16 23:19:45.000000000 -0400 Birth: 2026-09-16 23:19:44.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762665 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 23:19:50.000000000 -0400 Modify: 2026-09-16 23:19:50.000000000 -0400 Change: 2026-09-16 23:19:50.000000000 -0400 Birth: 2026-09-16 23:19:51.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281285 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 23:19:50.000000000 -0400 Modify: 2026-09-16 23:19:52.000000000 -0400 Change: 2026-09-16 23:19:52.000000000 -0400 Birth: 2026-09-16 23:19:50.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762666 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 23:19:57.000000000 -0400 Modify: 2026-09-16 23:19:59.000000000 -0400 Change: 2026-09-16 23:19:59.000000000 -0400 Birth: 2026-09-16 23:19:58.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281287 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 23:19:57.000000000 -0400 Modify: 2026-09-16 23:19:59.000000000 -0400 Change: 2026-09-16 23:19:59.000000000 -0400 Birth: 2026-09-16 23:19:58.000000000 -0400 PASS 54 (38s) == sanityn test 55a: rename vs unlink target dir ========= 23:20:12 (1789615212) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (15s) == sanityn test 55b: rename vs unlink source dir ========= 23:20:27 (1789615227) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 23:20:41 (1789615241) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 23:21:01 (1789615261) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 23:21:17 (1789615277) fail_loc=0x155 We survived after AB/BA race test fail_loc=0x0 PASS 55e (20s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 23:21:37 (1789615297) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 23:21:57 (1789615317) mdt.lustre-MDT0000.enable_rename_trylock=1 mdt.lustre-MDT0001.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (25s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 23:22:22 (1789615342) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789615344 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.465947 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (139s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 23:24:42 (1789615482) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789615487 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.841891 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (162s) == sanityn test 60: Verify data_version behaviour ======== 23:27:23 (1789615643) striped dir -i0 -c2 -H crush /mnt/lustre/d60.sanityn PASS 60 (13s) == sanityn test 70a: cd directory && rm directory ======== 23:27:36 (1789615656) PASS 70a (11s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 23:27:47 (1789615667) PASS 70b (13s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 23:28:00 (1789615680) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00835758 s, 7.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 23:28:04 (1789615684) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0261589 s, 2.5 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (6s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 23:28:10 (1789615690) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 23:28:14 (1789615694) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0185419 s, 3.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (5s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 23:28:19 (1789615699) llite.lustre-ffff9fe893100000.xattr_cache=1 llite.lustre-ffff9fe8a00ea000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 23:28:28 (1789615708) llite.lustre-ffff9fe893100000.xattr_cache=1 llite.lustre-ffff9fe8a00ea000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (8s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 23:28:35 (1789615715) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00993096 s, 103 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0322847 s, 31.7 kB/s 250256: get lock2 250258: get lock1 250258: done 250258: get lock2 250256: done 250256: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 250256: cannot get lock1: Resource deadlock avoided 250258: done 250258: exit rc=0 250256: done 250256: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 23:28:46 (1789615726) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0988249 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433605 s, 24.2 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 250848 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (14s) == sanityn test 76: Verify MDT open_files listing ======== 23:29:00 (1789615740) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (214s) == sanityn test 77a: check FIFO NRS policy =============== 23:32:34 (1789615954) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.528894 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836079 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142238 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169039 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152863 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156469 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202152 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227451 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303983 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377576 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309092 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401453 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310492 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386725 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470301 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416576 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.515561 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154676 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120456 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122137 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109784 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143017 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155713 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148843 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132492 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16977 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101246 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186407 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212161 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201225 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150676 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239565 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329634 s, 3.2 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (11s) == sanityn test 77b: check CRR-N NRS policy ============== 23:32:46 (1789615966) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.401858 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651997 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815072 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610835 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100202 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163535 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215511 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17074 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230239 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261053 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279429 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262139 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271844 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240213 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322991 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335476 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363027 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383876 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198866 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498986 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266609 s, 39.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00257734 s, 407 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855994 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00739205 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922732 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0062506 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844657 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776216 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062724 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724544 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572307 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351135 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487546 s, 21.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.485568 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419514 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107816 s, 9.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116264 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167109 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764702 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178923 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118567 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29043 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207827 s, 5.0 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235991 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20284 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282788 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35198 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383974 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33355 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347466 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824365 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10319 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824637 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903478 s, 11.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102987 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831273 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785446 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114104 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135133 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107022 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776711 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897666 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865189 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970927 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100878 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927149 s, 11.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (15s) == sanityn test 77c: check ORR NRS policy ================ 23:33:02 (1789615982) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.635673 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827043 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145333 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17297 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289166 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295596 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15152 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416245 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409489 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523651 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116965 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420292 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311487 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136167 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140537 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168766 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259166 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540583 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228059 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157161 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749369 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208441 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00669521 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165849 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152628 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724798 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815599 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187553 s, 55.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341899 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237856 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127303 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103711 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00945652 s, 111 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.871871 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821578 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135835 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145578 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255212 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168818 s, 6.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215531 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371232 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373555 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426053 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368364 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399574 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394909 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452328 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493133 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496699 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.628931 s, 1.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170424 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169405 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17206 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175723 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176059 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226847 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18635 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197657 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202573 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199213 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209002 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205494 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193965 s, 5.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225722 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229852 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251176 s, 4.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.00391 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11449 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15705 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118883 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282739 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317179 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330547 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10085 s, 10.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324426 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3859 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444717 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.54051 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471637 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.550798 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.656301 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.539995 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.824245 s, 1.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191862 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127983 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258473 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352428 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660414 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477166 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711073 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062148 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151763 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173263 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920721 s, 11.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260985 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116413 s, 90.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109165 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185513 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138809 s, 7.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (25s) == sanityn test 77d: check TRR nrs policy ================ 23:33:27 (1789616007) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.609804 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155051 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164112 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1727 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224917 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201185 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280148 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253825 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186936 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370226 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338078 s, 3.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346966 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367595 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394948 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44168 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332003 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169016 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612248 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109541 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136043 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249493 s, 420 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777067 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132962 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619722 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148681 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105386 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909308 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16374 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169838 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149747 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202326 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139635 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169616 s, 6.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.669013 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729759 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102649 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278714 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155664 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202187 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257724 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245741 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305876 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401253 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318258 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343466 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33957 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404811 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360321 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488126 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480696 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128535 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315529 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104532 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674654 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170494 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784832 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12448 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187515 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577626 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139114 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176365 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102375 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979173 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119904 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115094 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138075 s, 7.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.680639 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103537 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104224 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326902 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852724 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151946 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313269 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401742 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406214 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377161 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00976585 s, 107 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431567 s, 2.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43983 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122792 s, 85.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386083 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177038 s, 59.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00295978 s, 354 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472274 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471415 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125061 s, 83.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132649 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416671 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.532264 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447041 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214387 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166011 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07921 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015196 s, 69.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970675 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353727 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693053 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122514 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.582245 s, 1.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (24s) == sanityn test 77e: check TBF NID nrs policy ============ 23:33:51 (1789616031) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.16@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.395881 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113895 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176918 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192638 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303961 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20322 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269732 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340706 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392059 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300848 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387379 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452047 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483054 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381076 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44473 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429257 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.464657 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612653 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354377 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495625 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729714 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493864 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107718 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905002 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634081 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096649 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658242 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906035 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133116 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728071 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935026 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888741 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088632 s, 11.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.201039 s, 83.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443872 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772131 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21693 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220376 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186196 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337139 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334904 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317717 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366805 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331356 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376294 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405305 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373218 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364284 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370664 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43611 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973128 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593311 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586171 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337623 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906731 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979164 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125392 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136156 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106997 s, 98.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00859158 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00281669 s, 372 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125463 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128356 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953303 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246404 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511582 s, 20.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.785341 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763495 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119293 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31627 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259975 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20298 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311516 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.492896 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476867 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473821 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.509356 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399654 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465222 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.499058 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.524267 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.571577 s, 1.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404959 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101799 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359243 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120337 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428583 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086866 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110571 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826305 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114259 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00839612 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543613 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040025 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440933 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351286 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304679 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07409 s, 14.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.435655 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654284 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122671 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166782 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150241 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203367 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229415 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238795 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224795 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270966 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310858 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290168 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339867 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331437 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389827 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330745 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376533 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917552 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844176 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13434 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122859 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867673 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701961 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13267 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864436 s, 12.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120697 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094924 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102158 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101118 s, 10.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107872 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168996 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131856 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168307 s, 6.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (29s) == sanityn test 77f: check TBF JobID nrs policy ========== 23:34:20 (1789616060) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.60427 s, 27.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044058 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141196 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170901 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174945 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243092 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146932 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212621 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196168 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21672 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262644 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199572 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237275 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286835 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126551 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262072 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276475 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534007 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109126 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631285 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00603268 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278855 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153451 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147718 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133497 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135052 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138047 s, 7.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839572 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141761 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430937 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105093 s, 99.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161871 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913725 s, 11.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.663147 s, 25.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074175 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119477 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175546 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14021 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211054 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200584 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193248 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141801 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145447 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215518 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267803 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321607 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31271 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295654 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356374 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393174 s, 2.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027336 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00343559 s, 305 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683772 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483913 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356514 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495444 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418606 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993521 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808741 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562675 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106328 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537369 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114605 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067611 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878233 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719184 s, 14.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.409138 s, 41.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344376 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135909 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126843 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146267 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300726 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308441 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25563 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267478 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287306 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309213 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272128 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362986 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355732 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27926 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341937 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347317 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734308 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00294452 s, 356 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00266427 s, 394 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519715 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807997 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112674 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446637 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617485 s, 17.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775231 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370233 s, 28.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119547 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469844 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269601 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361316 s, 29.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264494 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344038 s, 30.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.742313 s, 22.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140287 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161839 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133433 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176459 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3577 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255714 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196477 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341541 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38807 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36357 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283661 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389367 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377773 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353209 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251239 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 5001+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326113 s, 3.2 MB/s [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450725 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447502 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660457 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220054 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00739503 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479444 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572853 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00476677 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103597 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00280083 s, 374 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00213647 s, 491 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378799 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00337046 s, 311 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101973 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490285 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451075 s, 232 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (27s) == sanityn test 77g: Change TBF type directly ============ 23:34:47 (1789616087) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (16s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 23:35:03 (1789616103) oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg216-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg216-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg216-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg216-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 PASS 77h (13s) == sanityn test 77i: Change rank of TBF rule ============= 23:35:16 (1789616116) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (27s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 23:35:43 (1789616143) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.11007 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126032 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241094 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272911 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292822 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35534 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.507344 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332527 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355728 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.541475 s, 1.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.648589 s, 1.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.581988 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.606086 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511044 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.639778 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.636244 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.554927 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084717 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171845 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06383 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032114 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150401 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107437 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812308 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917513 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12127 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122161 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136105 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091393 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113442 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116382 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104678 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188557 s, 5.6 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.0775 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7175 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (85s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 23:37:08 (1789616228) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg216-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg216-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg216-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg216-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.51621 s, 11.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125849 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131252 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12549 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203284 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277462 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196412 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210911 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277934 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253437 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35034 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378038 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326698 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362995 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422027 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369068 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416851 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985127 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855314 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639415 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120758 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136348 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819391 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133183 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135331 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161469 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152632 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169579 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188572 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193896 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220208 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183428 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188936 s, 5.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0377 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0155 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg216-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg216-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg216-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg216-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.34839 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901853 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200806 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840405 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181499 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231696 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260764 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218773 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196231 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198191 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261304 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24951 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271104 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299493 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359553 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298583 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271495 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532557 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748448 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633021 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672232 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520227 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921015 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679744 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708729 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524105 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606887 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108598 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119029 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770601 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794233 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902895 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101461 s, 10.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2655 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8953 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (164s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 23:39:52 (1789616392) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg216-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg216-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg216-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg216-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33524 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159719 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131539 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19551 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177524 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830232 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246153 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339487 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289251 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314815 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373748 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394116 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356626 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460749 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.510764 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474094 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462496 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884264 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578329 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129504 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775841 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829822 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806674 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124643 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155428 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988803 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165574 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115548 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195598 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200457 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141091 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184806 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160486 s, 6.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1117 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0272 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg216-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg216-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg216-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg216-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28187 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991316 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112978 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15229 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21527 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259044 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250953 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196692 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199186 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288513 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307838 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254588 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308202 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288501 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354839 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327855 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415598 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733998 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164876 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106168 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107007 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134119 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11392 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193035 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140997 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133724 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143673 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210472 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138758 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218962 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152805 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168744 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179111 s, 5.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0039 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8228 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (169s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 23:42:41 (1789616561) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.672781 s, 24.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065051 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118211 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18327 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197423 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195409 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168458 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309626 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269684 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266781 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282063 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36311 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336921 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364227 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38301 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236974 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266851 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598753 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638898 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636978 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180016 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488393 s, 21.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194639 s, 53.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179485 s, 58.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161394 s, 65.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182055 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073229 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10141 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00706696 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140502 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789198 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178219 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00973714 s, 108 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3316 s, 10.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1882 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.16@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.16@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.840135 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623412 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486524 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668987 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984656 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499386 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408316 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855574 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698297 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886217 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23818 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198464 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192374 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183972 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236523 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168832 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223476 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137296 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135861 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140786 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149441 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170842 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203859 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157174 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161073 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167942 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178524 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165783 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17767 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180311 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251181 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195525 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222802 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=2 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0131 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0187 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.16@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.631966 s, 26.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512506 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118175 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073902 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18276 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200052 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142273 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183343 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265193 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290538 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229199 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323316 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212808 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313116 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224153 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397206 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482672 s, 2.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165718 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115507 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193388 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101481 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203057 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557441 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204117 s, 5.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209758 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132533 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709008 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702185 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20474 s, 5.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079961 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738594 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247517 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840174 s, 12.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74216 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.46769 s, 19.2 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.61911 s, 27.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507952 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108283 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140943 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105234 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264224 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225701 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174304 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159273 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323827 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229543 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37019 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344387 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245417 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279563 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34034 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285146 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597685 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118377 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100748 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969249 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816438 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064276 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116995 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069887 s, 15.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880539 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124216 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981082 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878588 s, 11.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807599 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125776 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112702 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995537 s, 10.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.06 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.16 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3197 s, 8.5 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96632 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.00757 s, 16.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103333 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862856 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128641 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221373 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175759 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240247 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382577 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328122 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335357 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283679 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340625 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3382 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376422 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422741 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417872 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377126 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777437 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630556 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355093 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637766 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525545 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444569 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656656 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960769 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555948 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505328 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545799 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824011 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625946 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810703 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698072 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750041 s, 14.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8262 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6878 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.584163 s, 28.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110913 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126743 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195711 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164557 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232548 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248197 s, 4.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385146 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412284 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404886 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279439 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304818 s, 3.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422866 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379685 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456686 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30859 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485128 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106639 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064404 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11822 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120807 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836692 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11618 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951026 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157025 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154255 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098746 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129047 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121095 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138152 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128936 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111572 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115143 s, 9.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.36145 s, 14.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99108 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.765578 s, 21.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124722 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113961 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203231 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221178 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245903 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221481 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176494 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236675 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177083 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239964 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314374 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243889 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200542 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111506 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109577 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988642 s, 10.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116664 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707004 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11023 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973425 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135316 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128659 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185744 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129643 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136068 s, 7.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132816 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174246 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143188 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128765 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135987 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128365 s, 8.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2668 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88676 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7506 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92325 s, 10.6 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (465s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 23:50:26 (1789617026) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.583337 s, 28.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629008 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750025 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568346 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17396 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219047 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249995 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317691 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231805 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265036 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22381 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227404 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273093 s, 3.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264872 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324948 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293396 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304254 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00648598 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726602 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132707 s, 79.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777631 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897421 s, 11.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472791 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308337 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853709 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101624 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00737818 s, 142 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946647 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640706 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00767202 s, 137 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204256 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00632228 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00891777 s, 118 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.553 s, 13.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9285 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (52s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 23:51:18 (1789617078) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.16@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.16@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.914359 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106066 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964625 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134209 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288921 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281961 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312723 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418152 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346732 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521377 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494455 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501589 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.560573 s, 1.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480356 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.569566 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.573539 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.697379 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736777 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119557 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905903 s, 11.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110572 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949394 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961016 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185334 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118229 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117711 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13121 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193852 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168182 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170098 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173618 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178521 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211347 s, 5.0 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.16946 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92481 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (56s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 23:52:14 (1789617134) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.16@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.798164 s, 21.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659314 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865666 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178753 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178705 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24121 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261303 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250782 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210086 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.524956 s, 2.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494559 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484668 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.565959 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.532862 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.60797 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299765 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299101 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191345 s, 54.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282421 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291197 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179012 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.590346 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332183 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313878 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333031 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313562 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321554 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340414 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213195 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359098 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36075 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120391 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.645843 s, 1.6 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.64782 s, 13.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.75565 s, 18.2 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (49s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 23:53:03 (1789617183) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.771628 s, 21.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586528 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108017 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584907 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115522 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1534 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297034 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252527 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385987 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310411 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390818 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397609 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437977 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435295 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460217 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407345 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.517492 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120486 s, 87.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224725 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088431 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639295 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046208 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456175 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027531 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434032 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366819 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430355 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474502 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054778 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063287 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065155 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704758 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988592 s, 10.6 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.981 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86733 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.74358 s, 15.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90853 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (100s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 23:54:43 (1789617283) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.15997 s, 14.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671648 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103429 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968245 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240656 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270082 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294067 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30706 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296931 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222629 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318758 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292055 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284799 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367045 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368735 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277745 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374238 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732129 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063588 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992794 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07459 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106346 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113785 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150185 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107159 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078693 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12114 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130188 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126576 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124674 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110153 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131565 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147817 s, 7.1 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0665 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0027 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (78s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 23:56:01 (1789617361) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.849709 s, 19.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103582 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143858 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829159 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180714 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122598 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109883 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210034 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129781 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25488 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342648 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305882 s, 3.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fe8a00ea000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425364 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237454 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292295 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.505057 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444351 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9fe8a00ea000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04721 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062424 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796851 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919078 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126172 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102794 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938557 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112335 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100191 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957074 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117695 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892355 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968253 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127897 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133802 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120499 s, 8.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.0833 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88179 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg216-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.867514 s, 19.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105709 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091086 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177307 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203815 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160748 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205564 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156669 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237754 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17327 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116914 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232944 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331374 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267072 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329559 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271729 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305981 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234313 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921053 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946808 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029182 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379626 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255762 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352807 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00532481 s, 197 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520809 s, 20.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00741867 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502136 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577139 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408077 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120825 s, 86.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216229 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105531 s, 9.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5551 s, 8.4 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0213 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7493 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98069 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (156s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 23:58:37 (1789617517) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:) Stopping client oleg216-client.virtnet /mnt/lustre opts: Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg216-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg216-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg216-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.16,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg216-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.16,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg216-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.16,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70382/testresults/sanityn-zfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.16,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg216-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: kmemleak disabled oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: detected 4 online CPUs by sysfs oleg216-server: MODOPTS_LIBCFS= oleg216-server: Force libcfs to create 2 CPU partitions oleg216-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg216-server: oleg216-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre Starting client oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre Started clients oleg216-client.virtnet: 192.168.202.116@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 Starting client oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 Started clients oleg216-client.virtnet: 192.168.202.116@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fe884a41800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9fe887e6c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fe884a41800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fe887e6c800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.685751 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479899 s, 21.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12906 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560198 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896629 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705459 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19189 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679888 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570192 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079445 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296779 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323463 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330569 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342946 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303941 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345091 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377032 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fe884a41800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354151 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421126 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432861 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363397 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370198 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415694 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401798 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412358 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408765 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432931 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435863 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40236 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397555 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477457 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420978 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426878 s, 2.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.76025 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.41074 s, 19.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.584835 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057306 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824616 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791113 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101003 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129448 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141129 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175306 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161628 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241184 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252765 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270296 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182714 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219364 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250508 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276584 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299257 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe884a41800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9fe884a41800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413618 s, 254 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121532 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01324 s, 79.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00841172 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537506 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0081914 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642645 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220876 s, 47.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853823 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225865 s, 46.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12681 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094947 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169357 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947521 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968989 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466698 s, 22.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.36028 s, 14.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92623 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (250s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 00:02:47 (1789617767) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg216-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 00:03:05 (1789617785) On MGS 192.168.202.116, active = nodemap.active=1 On MGS 192.168.202.116, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.633028 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103915 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917401 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839498 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104404 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145985 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219648 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26535 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313446 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392538 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398851 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419314 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431462 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463686 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387035 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396662 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468368 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe884a41800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9fe884a41800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237707 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214657 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21765 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221371 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237636 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33932 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198842 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315276 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269946 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254617 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30171 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193657 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239874 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276627 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266595 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237483 s, 4.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.86504 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.29131 s, 19.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.202.116, TBF.id = On MGS 192.168.202.116, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.729072 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100459 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108139 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213994 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229064 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218756 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196508 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277593 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283489 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351513 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312717 s, 3.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320262 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394036 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353042 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392189 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392028 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476728 s, 2.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9fe884a41800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100594 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463215 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122222 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11401 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116326 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551615 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494069 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123799 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572261 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00709551 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721166 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162625 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127209 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00699932 s, 150 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00785229 s, 134 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261567 s, 40.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.05883 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.10553 s, 33.8 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg216-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg216-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 On MGS 192.168.202.116, active = nodemap.active=1 On MGS 192.168.202.116, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.837242 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062055 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593438 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894829 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139044 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126091 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19113 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258588 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272043 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218193 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319495 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228684 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24697 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362742 s, 2.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277423 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306913 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305017 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe884a41800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614614 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290383 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075455 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440447 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09175 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647967 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815442 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599591 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850773 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640754 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802017 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728241 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061446 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762219 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114098 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102989 s, 10.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1415 s, 10.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92668 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.202.116, TBF.id = On MGS 192.168.202.116, active = nodemap.active=0 PASS 77kj (184s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 00:06:09 (1789617969) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (12s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 00:06:21 (1789617981) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg216-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.60423 s, 291 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg216-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0643 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 00:07:19 (1789618039) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.588465 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111826 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065652 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681885 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146946 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237003 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238071 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30248 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341779 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358989 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361254 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41249 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36596 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410618 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471516 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468838 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.531161 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9fe884a41800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fe884a41800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00486878 s, 215 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876511 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00632225 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520463 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146031 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784885 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049277 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129783 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049391 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103911 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280687 s, 37.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932085 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030906 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104485 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083535 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00951607 s, 110 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg216-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg216-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.274 s, 14.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.44173 s, 19.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.646489 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771523 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939572 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164318 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707085 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195367 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193251 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270916 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25686 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212015 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273982 s, 3.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238297 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321924 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279565 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318568 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322023 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324733 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662843 s, 15.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670846 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714422 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760132 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708141 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13936 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973042 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109673 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111473 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103453 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135599 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131439 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130539 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144108 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147863 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151664 s, 6.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg216-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg216-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.23079 s, 12.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.51971 s, 19.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (94s) == sanityn test 77o: Changing rank should not panic ====== 00:08:53 (1789618133) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (14s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 00:09:07 (1789618147) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (115s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 00:11:02 (1789618262) mds.MDS.mdt.nrs_policies=tbf oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg216-client: oleg216-server: ssh exited with exit code 36 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 oleg216-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg216-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg216-client: oleg216-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (37s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 00:11:40 (1789618300) striped dir -i0 -c1 -H crush /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 294835 kill md_thread xargs PASS 77r (53s) == sanityn test 77s: Check TBF LRU shrinker ============== 00:12:33 (1789618353) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (55s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 00:13:28 (1789618408) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (8s) == sanityn test 79: xattr: intent error ================== 00:13:36 (1789618416) striped dir -i1 -c2 -H all_char /mnt/lustre/d79.sanityn File: /mnt/lustre/d79.sanityn Size: 32768 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129637319639057 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-17 00:13:38.000000000 -0400 Modify: 2026-09-17 00:13:38.000000000 -0400 Change: 2026-09-17 00:13:38.000000000 -0400 Birth: 2026-09-17 00:13:39.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (18s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 00:13:55 (1789618435) total: 10 open/close in 0.20 seconds: 49.16 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 Finish migration, then checking.. PASS 80a (10s) == sanityn test 80b: Accessing directory during migration ========================================================== 00:14:05 (1789618445) start migration thread 302069 accessing the migrating directory for 1 minute... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete concurrent access 102 failures, 348 successes PASS 80b (92s) == sanityn test 81a: rename and stat under striped directory ========================================================== 00:15:38 (1789618538) stat: cannot statx '/mnt/lustre/d81a.sanityn/d0/0': No such file or directory File: /mnt/lustre/d81a.sanityn/d0/0 Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272381367833 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-17 00:15:40.000000000 -0400 Modify: 2026-09-17 00:15:40.000000000 -0400 Change: 2026-09-17 00:15:40.000000000 -0400 Birth: 2026-09-17 00:15:40.000000000 -0400 PASS 81a (8s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 00:15:45 (1789618545) - open/close 813 (time 1789618559.88 total 10.00 last 81.29) - open/close 1631 (time 1789618569.94 total 20.05 last 81.36) - open/close 2465 (time 1789618579.96 total 30.08 last 83.23) total: 2500 open/close in 30.57 seconds: 81.78 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 304329 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (483s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 00:23:48 (1789619028) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (5s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 00:23:53 (1789619033) - open/close 607 (time 1789619050.41 total 10.01 last 60.64) - open/close 1247 (time 1789619060.41 total 20.01 last 63.98) - open/close 2000 (time 1789619070.41 total 30.01 last 75.30) total: 2500 open/close in 36.35 seconds: 68.78 ops/second - open/close 740 (time 1789619092.02 total 10.01 last 73.94) - open/close 1494 (time 1789619102.03 total 20.02 last 75.33) - open/close 2219 (time 1789619112.04 total 30.03 last 72.41) total: 2500 open/close in 33.96 seconds: 73.63 ops/second snapshot_time 1789619265.508285241 secs.nsecs start_time 1789619037.455135923 secs.nsecs elapsed_time 228.053149318 secs.nsecs open 2505 samples [usecs] 157 19817 3412245 9494669561 close 2505 samples [usecs] 129 8457 912242 644638744 mknod 2501 samples [usecs] 378 19697 2847804 7072464342 unlink 2500 samples [usecs] 555 103415 6883823 109961748409 mkdir 2 samples [usecs] 1272 28539 29811 816092505 rmdir 2 samples [usecs] 6338 11301 17639 167882845 rename 2500 samples [usecs] 391 66491 20911694 402185311918 getattr 3773 samples [usecs] 1 32349 320918 1707353428 statfs 44 samples [usecs] 10 88 1053 31795 parallel_rename_file 2500 samples [usecs] 391 66491 20911694 402185311918 crossdir_rename 2500 samples [usecs] 391 66491 20911694 402185311918 readpage 28 samples [usecs] 8352 63570 873000 34590730292 snapshot_time 1789619265.508375181 secs.nsecs start_time 1789619037.455194972 secs.nsecs elapsed_time 228.053180209 secs.nsecs open 2504 samples [usecs] 123 21618 3225967 7759667093 close 2504 samples [usecs] 91 10245 949548 901176976 mknod 2500 samples [usecs] 425 21500 2709440 5677577646 unlink 2500 samples [usecs] 494 145364 7816616 179845363116 mkdir 2 samples [usecs] 4991 29795 34786 912652106 rmdir 2 samples [usecs] 7130 69068 76198 4821225524 rename 2500 samples [usecs] 423 41473 11194335 97871078353 getattr 3764 samples [usecs] 7 17116 238514 789817600 statfs 44 samples [usecs] 11 2993 4195 9113245 parallel_rename_file 2500 samples [usecs] 423 41473 11194335 97871078353 crossdir_rename 2500 samples [usecs] 423 41473 11194335 97871078353 readpage 31 samples [usecs] 5766 74622 1092835 49444392811 PASS 81d (237s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 00:27:50 (1789619270) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.8439 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.8439 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 00:27:57 (1789619277) start pid 318435 to create/unlink striped directory start pid 318436 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6198: 318435 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6198: 318436 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (126s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 00:30:03 (1789619403) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 00:30:16 (1789619416) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789619418 Finishing test test1 at 1789619420 PASS 85 (9s) == sanityn test 90: open/create and unlink striped directory ========================================================== 00:30:25 (1789619425) start pid 323291 to open/create under striped directory start pid 323292 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6265: 323291 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6265: 323292 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (187s) == sanityn test 91: chmod and unlink striped directory === 00:33:32 (1789619612) start pid 325728 to chmod striped directory start pid 325729 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6302: 325728 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6302: 325729 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (187s) == sanityn test 92: create remote directory under orphan directory ========================================================== 00:36:39 (1789619799) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (8s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 00:36:47 (1789619807) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 3558 0xde6 0x2c0000400 0 3594 0xe0a 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5446 0x1546 0x280000401 1 5440 0x1540 0x2c0000401 rc1=0 and rc2=0 PASS 93 (20s) == sanityn test 94: signal vs CP callback race =========== 00:37:07 (1789619827) osc.lustre-OST0000-osc-ffff9fe884a41800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9fe887e6c800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9fe884a41800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9fe887e6c800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.377816 s, 21.7 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.96808 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 330355 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.090817 s, 45.1 kB/s osc.lustre-OST0000-osc-ffff9fe884a41800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9fe887e6c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fe884a41800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fe887e6c800.idle_timeout=debug PASS 94 (19s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 00:37:26 (1789619846) llite.lustre-ffff9fe884a41800.fast_read=0 llite.lustre-ffff9fe887e6c800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110405 s, 19.0 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9fe884a41800.fast_read=1 llite.lustre-ffff9fe887e6c800.fast_read=1 PASS 95a (22s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 00:37:48 (1789619868) llite.lustre-ffff9fe884a41800.fast_read=0 llite.lustre-ffff9fe887e6c800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00615793 s, 2.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9fe884a41800.fast_read=1 llite.lustre-ffff9fe887e6c800.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 00:38:01 (1789619881) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 00:38:05 (1789619885) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00749293 s, 547 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 00:38:12 (1789619892) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.196939 s, 10.6 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 00:38:20 (1789619900) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189431 s, 11.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (10s) == sanityn test 100e: DoM: read on open and file size ==== 00:38:30 (1789619910) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00744651 s, 1.6 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 00:38:38 (1789619918) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0175332 s, 234 kB/s PASS 101a (9s) == sanityn test 101b: Discard DoM data on rename ========= 00:38:47 (1789619927) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0377835 s, 108 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 00:38:55 (1789619935) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0137253 s, 298 kB/s PASS 101c (10s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 00:39:05 (1789619945) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 f6 17 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0x17f6:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: d0 0b 00 40 02 00 00 00 ce 13 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd0:0x13ce:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 f7 17 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0x17f7:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: d0 0b 00 40 02 00 00 00 cf 13 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd0:0x13cf:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: d0 0b 00 40 02 00 00 00 d0 13 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd0:0x13d0:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102 (12s) == sanityn test 103: Test size correctness with lockahead ========================================================== 00:39:17 (1789619957) striped dir -i1 -c2 -H crush /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789619961 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789619964 Finishing test test23 at 1789619964 PASS 103 (16s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 00:39:33 (1789619973) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 00:39:38 (1789619978) striped dir -i1 -c2 -H crush /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 00:40:07 (1789620007) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 00:40:10 (1789620010) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0079769 s, 128 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381372410 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-17 00:40:11.000000000 -0400 Modify: 2026-09-17 00:40:11.000000000 -0400 Change: 2026-09-17 00:40:11.000000000 -0400 Birth: 2026-09-17 00:40:12.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272381372410 -rw-r--r-- 1 0 0 1789620012 1789620011 1789620011 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381372410 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-17 00:40:11.000000000 -0400 Modify: 2026-09-17 00:40:11.000000000 -0400 Change: 2026-09-17 00:40:11.000000000 -0400 Birth: 2026-09-17 00:40:12.000000000 -0400 1789620011 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 00:40:17 (1789620017) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 00:40:25 (1789620025) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.31658 s, 33.1 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5453 0x154d 0x280000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 First grouplock blocks second one PASS 107a (8s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 00:40:34 (1789620034) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.353708 s, 29.6 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5446 0x1546 0x2c0000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 00:40:47 (1789620047) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0070805 s, 1.4 MB/s llite.lustre-ffff9fe884a41800.hybrid_io=0 llite.lustre-ffff9fe887e6c800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.70697 s, 1.8 MB/s PASS 108a (11s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 00:40:58 (1789620058) 192.168.202.116@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg216-client.virtnet /mnt/lustre (opts:) 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.116@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg216-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: kmemleak disabled oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: detected 4 online CPUs by sysfs oleg216-server: MODOPTS_LIBCFS= oleg216-server: Force libcfs to create 2 CPU partitions oleg216-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 192.168.202.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.116@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg216-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg216-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: kmemleak disabled oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: detected 4 online CPUs by sysfs oleg216-server: MODOPTS_LIBCFS= oleg216-server: Force libcfs to create 2 CPU partitions oleg216-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre Starting client: oleg216-client.virtnet: -o user_xattr,flock 192.168.202.116@tcp:/lustre /mnt/lustre2 PASS 109 (74s) == sanityn test 110: do not grant another lock on resend ========================================================== 00:42:12 (1789620132) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272431697923 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-17 00:42:14.000000000 -0400 Modify: 2026-09-17 00:42:14.000000000 -0400 Change: 2026-09-17 00:42:14.000000000 -0400 Birth: 2026-09-17 00:42:15.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg216-client.virtnet oleg216-client.virtnet PASS 110 (80s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 00:43:32 (1789620212) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 7188: 345958 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (14s) == sanityn test 112: update max-inherit in default LMV === 00:43:46 (1789620226) lmv.lustre-clilmv-ffff9fe8814d1000.qos_threshold_rr=90 lmv.lustre-clilmv-ffff9fe885084800.qos_threshold_rr=90 PASS 112 (14s) == sanityn test 113: check servers of specified fs ======= 00:44:00 (1789620240) lustre-OST0000-osc-ffff9fe8814d1000 active. lustre-OST0001-osc-ffff9fe8814d1000 active. lustre-OST0000-osc-ffff9fe8814d1000 active. lustre-OST0001-osc-ffff9fe8814d1000 active. MGC192.168.202.116@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 00:44:08 (1789620248) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (26s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 00:44:34 (1789620274) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 00:44:38 (1789620278) llite.lustre-ffff9fe8814d1000.intent_mkdir=1 llite.lustre-ffff9fe885084800.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000013a0:0x10c:0x0] 1 [0x240000bd2:0x100:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none PASS 116 (6s) == sanityn test 121: trunc append race =================== 00:44:44 (1789620284) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0166201 s, 1.2 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.091955 s, 223 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 00:44:54 (1789620294) PASS 122 (17s) == sanityn test 200: service remains healthy while able to process request ========================================================== 00:45:11 (1789620311) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 striped dir -i0 -c2 -H crush2 /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.186 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2687 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2475 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2244 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2578 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2382 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3174 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2807 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.343 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3285 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3055 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3288 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3273 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3376 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3252 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4837 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4311 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4799 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg216-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9fe8814d1000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fe8814d1000.ost_server_uuid in FULL state after 0 sec PASS 200 (82s) cleanup: ====================================================== == sanityn test complete, duration 12712 sec ============= 00:46:35 (1789620395) === sanityn: start cleanup 00:46:37 (1789620397) === Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg216-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 00:51:19 (1789620679) ===