-----============= acceptance-small: sanityn ============----- Fri Oct 2 01:41:40 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITYN_ONLY='' SANITYN_EXCEPT='' oleg318-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 56b skipping tests SLOW=no: 33a === sanityn: start setup 01:42:01 (1790919721) === Starting client oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Mount client oleg318-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Started clients oleg318-client.virtnet: 192.168.203.118@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg318-client.virtnet: executing check_config_client /mnt/lustre oleg318-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg318-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d958326b000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d959031d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d958326b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d959031d800.idle_timeout=debug disable quota as required oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 01:42:33 (1790919753) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188425 s, 55.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8439] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 01:42:36 (1790919756) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 01:42:43 (1790919763) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (5s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 01:42:48 (1790919768) -rw-r--r-- 1 root root 0 Oct 2 01:42 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (5s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 01:42:53 (1790919773) -rw-r--r-- 1 root root 0 Oct 2 01:42 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (5s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 01:42:58 (1790919778) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (5s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 01:43:03 (1790919783) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 01:43:10 (1790919790) total 1 -rwxrwxrwx 1 root root 0 Oct 2 01:42 f2a -rwxrwxrwx 1 root root 0 Oct 2 01:42 f2b total 1 -rwxrwxrwx 1 root root 0 Oct 2 01:42 f2a -rwxrwxrwx 1 root root 0 Oct 2 01:42 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0112024 s, 45.7 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 01:43:16 (1790919796) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (7s) == sanityn test 2g: check blocks update on sync write ==== 01:43:23 (1790919803) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.389535 s, 5.4 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 01:43:30 (1790919810) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 01:43:35 (1790919815) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 01:43:41 (1790919821) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 01:43:47 (1790919827) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 01:43:52 (1790919832) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 01:43:58 (1790919838) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 01:44:05 (1790919845) PASS 9a (9s) == sanityn test 9b: append to striped sparse file ======== 01:44:14 (1790919854) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770958 s, 13.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0342681 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 01:44:22 (1790919862) 1+0 records in 1+0 records out 1 byte copied, 0.0123095 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0130829 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0181564 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0174729 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0257651 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0132577 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0150471 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0186112 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011744 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010551 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.022986 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0104141 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 01:44:30 (1790919870) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0174128 s, 176 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.104587 s, 39.2 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00172682 s, 1.8 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 01:44:37 (1790919877) striped dir -i1 -c2 -H crush /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 304: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 01:44:44 (1790919884) warning: '-runtime' deprecated, use '-t runtime' instead - link 423 (time 1790919896.22 total 10.01 last 42.27) - link 750 (time 1790919906.24 total 20.02 last 32.66) - link 1088 (time 1790919916.26 total 30.05 last 33.71) - link 1445 (time 1790919926.29 total 40.07 last 35.60) - link 1794 (time 1790919936.29 total 50.08 last 34.89) - link 2149 (time 1790919946.30 total 60.09 last 35.46) - link 2537 (time 1790919956.31 total 70.09 last 38.78) - link 2909 (time 1790919966.34 total 80.13 last 37.07) - link 3260 (time 1790919976.35 total 90.13 last 35.08) total: 3571 link in 100.01 seconds: 35.71 ops/second using seed 2689437429 running for 100 seconds - stat 8917 (time 1790919898 ; total 11 ; last 11) - stat 10000 (time 1790919899 ; total 12 ; last 1) - stat 17339 (time 1790919910 ; total 23 ; last 11) - stat 20000 (time 1790919913 ; total 26 ; last 3) - stat 27992 (time 1790919924 ; total 37 ; last 11) - stat 30000 (time 1790919927 ; total 40 ; last 3) - stat 38774 (time 1790919938 ; total 51 ; last 11) - stat 40000 (time 1790919942 ; total 55 ; last 4) - stat 43767 (time 1790919953 ; total 66 ; last 11) - stat 47588 (time 1790919964 ; total 77 ; last 11) - stat 50000 (time 1790919972 ; total 85 ; last 8) - stat 53553 (time 1790919983 ; total 96 ; last 11) total: 54830 stats in 100 seconds: 548.299988 stats/second - unlinked 0 (time 1790919937 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3571) error: No such file or directory total: 3570 unlinks in 89 seconds: 40.112358 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22817) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22819) - No such process PASS 12 (146s) == sanityn test 13: test directory page revocation ======= 01:47:10 (1790920030) striped dir -i1 -c2 -H all_char /mnt/lustre/d13 f13 f13 PASS 13 (7s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 01:47:17 (1790920037) striped dir -i0 -c2 -H all_char /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 339: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 24028 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 01:47:25 (1790920045) striped dir -i0 -c2 -H crush /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 24619 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 01:47:32 (1790920052) striped dir -i0 -c2 -H crush /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 354: 25209 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 01:47:38 (1790920058) striped dir -i0 -c2 -H all_char /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 366: 25804 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 01:47:45 (1790920065) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 377: 26399 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 01:47:53 (1790920073) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7530496 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7530496KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (49s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 01:48:43 (1790920123) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.564377 s, 14.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.448698 s, 18.7 MB/s Chance of close/open is 1 in 50 Seed set to 127 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1790920128.791114 write 0x4ff301 thru 0x502bd7 (0x38d7 bytes) 000002[1] 1790920128.802793 write 0x1bef6b thru 0x1c7452 (0x84e8 bytes) 000003[1] 1790920128.813867 punch from 0x0ffd0f to 0x10464d (0x493f bytes) 000004[0] 1790920128.829927 write 0x19d0da thru 0x1a1084 (0x3fab bytes) 000005[1] 1790920128.872915 punch from 0x0fa03d to 0x107682 (0xd646 bytes) 000008[1] 1790920128.928288 mapread 0x0acc1f thru 0x0bc5d7 (0xf9b9 bytes) 000010[0] 1790920129.026317 read 0x495adb thru 0x4998c6 (0x3dec bytes) 000011[0] 1790920129.068194 write 0x177dbc thru 0x18522a (0xd46f bytes) 000012[1] 1790920129.106687 write 0x50c184 thru 0x5153a6 (0x9223 bytes) truncating to largest ever: 0x683201 000015[0] 1790920129.164820 trunc from 0x5153a7 to 0x683200 (0x16de5a bytes) 000017[1] 1790920129.235336 punch from 0x5d0c81 to 0x5dbd06 (0xb086 bytes) 000018[0] 1790920129.262116 write 0x7d9085 thru 0x7e2a55 (0x99d1 bytes) 000019[0] 1790920129.296354 mapread 0x625e75 thru 0x62e2e3 (0x846f bytes) 000020[1] 1790920129.341048 write 0x28ab2a thru 0x28bf86 (0x145d bytes) 000022[1] 1790920129.401135 read 0x2276ca thru 0x22bfde (0x4915 bytes) 000025[1] 1790920129.471058 mapwrite 0x350b37 thru 0x3599a3 (0x8e6d bytes) 000026[1] 1790920129.590059 punch from 0x4db236 to 0x4dbc75 (0xa40 bytes) 000028[1] 1790920129.671210 write 0x21c86a thru 0x21f0cd (0x2864 bytes) 000029[1] 1790920129.683337 mapread 0x02c84f thru 0x02f781 (0x2f33 bytes) 000030[1] 1790920129.706118 write 0x04cd62 thru 0x05c1f6 (0xf495 bytes) 000031[0] 1790920129.718829 read 0x5be7f3 thru 0x5c02c9 (0x1ad7 bytes) 000032[1] 1790920129.767895 mapread 0x32b2b8 thru 0x32bf11 (0xc5a bytes) 000033[1] 1790920129.804009 mapread 0x177075 thru 0x17b3a7 (0x4333 bytes) 000034[0] 1790920129.871803 mapwrite 0x053f4b thru 0x0617a7 (0xd85d bytes) 000035[0] 1790920130.074957 mapwrite 0x415765 thru 0x420a89 (0xb325 bytes) 000037[1] 1790920130.292148 mapread 0x78369b thru 0x79062c (0xcf92 bytes) 000038[1] 1790920130.363141 mapread 0x501913 thru 0x5110d8 (0xf7c6 bytes) 000039[1] 1790920130.473238 mapwrite 0x3930b9 thru 0x397f59 (0x4ea1 bytes) 000040[0] 1790920130.601255 punch from 0x758c88 to 0x766792 (0xdb0b bytes) 000041[1] 1790920130.633492 mapread 0x0f6e2e thru 0x10297f (0xbb52 bytes) 000043[1] 1790920130.742551 punch from 0x520636 to 0x528f58 (0x8923 bytes) 000048[1] 1790920130.935161 punch from 0x18b42e to 0x18cc9c (0x186f bytes) 000049[1] 1790920130.941156 mapread 0x4b4e7a thru 0x4bbd25 (0x6eac bytes) 000051[1] 1790920130.970222 mapread 0x782ac7 thru 0x787b24 (0x505e bytes) truncating to largest ever: 0x706c63 000054[1] 1790920131.038129 trunc from 0x7e2a56 to 0x706c62 (0xdbdf3 bytes) 000055[0] 1790920131.078252 punch from 0x47dc14 to 0x47f824 (0x1c11 bytes) 000058[0] 1790920131.181197 mapread 0x368851 thru 0x36d7e3 (0x4f93 bytes) 000059[0] 1790920131.235336 punch from 0x4279fa to 0x429de4 (0x23eb bytes) 000060[1] 1790920131.252865 read 0x686215 thru 0x68da74 (0x7860 bytes) 000061[1] 1790920131.275862 write 0x42a27e thru 0x432134 (0x7eb7 bytes) 000063[1] 1790920131.296257 write 0x42b0f5 thru 0x432171 (0x707d bytes) 000064[0] 1790920131.308443 mapwrite 0x420e06 thru 0x42940b (0x8606 bytes) 000065[0] 1790920131.497627 mapread 0x33bd8a thru 0x343116 (0x738d bytes) 000066[0] 1790920131.513371 read 0x52bb08 thru 0x52cd24 (0x121d bytes) 000067[0] 1790920131.527687 punch from 0x0ef9f4 to 0x0f9ed6 (0xa4e3 bytes) 000070[0] 1790920131.622833 write 0x3c7ff8 thru 0x3d7bff (0xfc08 bytes) 000072[0] 1790920131.664181 punch from 0x4be4b2 to 0x4cb0d4 (0xcc23 bytes) 000075[0] 1790920131.734286 read 0x2cc327 thru 0x2d8a4b (0xc725 bytes) 000077[1] 1790920131.775551 mapwrite 0x080177 thru 0x085ded (0x5c77 bytes) 000078[1] 1790920132.051147 punch from 0x0339b0 to 0x03cf45 (0x9596 bytes) 000079[1] 1790920132.087517 mapread 0x26d83e thru 0x27ae70 (0xd633 bytes) 000080[0] 1790920132.196091 punch from 0x2e73fc to 0x2f6ed2 (0xfad7 bytes) 000081[0] 1790920132.237078 trunc from 0x706c63 to 0x241f71 (0x4c4cf1 bytes) 000085[0] 1790920132.351173 mapread 0x0f2243 thru 0x0f9a5b (0x7819 bytes) 000087[1] 1790920132.446975 write 0x584c79 thru 0x5906b9 (0xba41 bytes) 000088[0] 1790920132.474831 read 0x35b6a8 thru 0x360a4e (0x53a7 bytes) 000089[0] 1790920132.573056 mapread 0x12439f thru 0x1294aa (0x510c bytes) truncating to largest ever: 0x9f1884 000090[1] 1790920132.600190 trunc from 0x5906ba to 0x9f1883 (0x4611ca bytes) 000096[0] 1790920132.672180 mapread 0x74b398 thru 0x74cc3d (0x18a6 bytes) 000097[0] 1790920132.751093 write 0x7d9662 thru 0x7e911f (0xfabe bytes) 000098[1] 1790920132.804994 punch from 0x7f0cf6 to 0x7f2157 (0x1462 bytes) 000099[1] 1790920132.865285 trunc from 0x9f1884 to 0x81401c (0x1dd867 bytes) 000100[0] 1790920132.913776 trunc from 0x81401d to 0x43dd7c (0x3d62a0 bytes) 000102[0] 1790920133.058090 punch from 0x2ad51a to 0x2b21d1 (0x4cb8 bytes) 000103[0] 1790920133.069857 mapwrite 0x3025bb thru 0x306425 (0x3e6b bytes) 000104[0] 1790920133.169997 write 0x5ebfe6 thru 0x5f69d3 (0xa9ee bytes) 000105[1] 1790920133.173865 read 0x188209 thru 0x196b53 (0xe94b bytes) 000106[0] 1790920133.222695 read 0x510aaa thru 0x51a8c1 (0x9e18 bytes) 000108[1] 1790920133.267616 read 0x000c20 thru 0x000e99 (0x27a bytes) 000109[0] 1790920133.304720 read 0x022575 thru 0x02460d (0x2099 bytes) 000111[1] 1790920133.335646 punch from 0x4be48b to 0x4c2147 (0x3cbd bytes) 000113[0] 1790920133.360333 write 0x543a05 thru 0x54852e (0x4b2a bytes) 000114[0] 1790920133.394718 mapwrite 0x96de62 thru 0x978504 (0xa6a3 bytes) 000116[1] 1790920133.518178 read 0x13660f thru 0x142612 (0xc004 bytes) 000117[0] 1790920133.541107 trunc from 0x978505 to 0x01c2d0 (0x95c234 bytes) 000119[1] 1790920133.595051 mapwrite 0x0a72cc thru 0x0b2990 (0xb6c5 bytes) 000120[0] 1790920133.756818 mapwrite 0x1437da thru 0x15053a (0xcd61 bytes) 000121[1] 1790920133.925073 mapread 0x08db8e thru 0x08e706 (0xb79 bytes) 000122[0] 1790920133.986180 read 0x13101b thru 0x13eaf4 (0xdada bytes) 000123[1] 1790920134.047814 trunc from 0x15053b to 0x37ce67 (0x22c92d bytes) 000124[1] 1790920134.142375 mapwrite 0x0b2b7e thru 0x0ba3f4 (0x7877 bytes) 000127[0] 1790920134.276753 read 0x0c7436 thru 0x0cc0a4 (0x4c6f bytes) 000128[1] 1790920134.398815 read 0x078bbf thru 0x080b71 (0x7fb3 bytes) 000129[1] 1790920134.482691 mapwrite 0x1f34b5 thru 0x1f9e98 (0x69e4 bytes) 000130[1] 1790920134.658937 mapread 0x1c7885 thru 0x1d0cbb (0x9437 bytes) 000131[0] 1790920134.799730 trunc from 0x37ce68 to 0x7dc6eb (0x45f884 bytes) 000133[1] 1790920134.986240 trunc from 0x7dc6ec to 0x4f12b4 (0x2eb437 bytes) 000134[0] 1790920135.066188 punch from 0x3fd5cd to 0x3ff96d (0x23a1 bytes) 000135[0] 1790920135.123283 trunc from 0x4f12b5 to 0x8310ea (0x33fe36 bytes) 000137[0] 1790920135.240712 mapwrite 0x1559ad thru 0x162ce0 (0xd334 bytes) 000141[0] 1790920135.483268 trunc from 0x8310eb to 0x3f8b05 (0x4385e5 bytes) 000142[1] 1790920135.579654 trunc from 0x3f8b06 to 0x93b660 (0x542b5b bytes) 000143[0] 1790920135.699570 mapread 0x7983c1 thru 0x7a1c5d (0x989d bytes) 000144[0] 1790920135.942193 trunc from 0x93b661 to 0x7ee244 (0x14d41c bytes) 000145[1] 1790920136.091044 read 0x689841 thru 0x694710 (0xaed0 bytes) 000146[0] 1790920136.127860 write 0x292758 thru 0x295163 (0x2a0c bytes) 000149[0] 1790920136.194449 write 0x431ea8 thru 0x43f4fc (0xd655 bytes) 000151[1] 1790920136.240066 mapread 0x4ff3b0 thru 0x50c991 (0xd5e2 bytes) 000153[1] 1790920136.296329 write 0x890a3e thru 0x89f7c8 (0xed8b bytes) 000154[1] 1790920136.313271 write 0x7870bc thru 0x795706 (0xe64b bytes) 000155[1] 1790920136.361134 mapwrite 0x49c336 thru 0x49c561 (0x22c bytes) 000158[1] 1790920136.452359 punch from 0x1e4742 to 0x1e9eb2 (0x5771 bytes) 000162[0] 1790920136.458516 trunc from 0x89f7c9 to 0x76a605 (0x1351c3 bytes) 000164[0] 1790920136.539279 write 0x55b4d7 thru 0x561f1d (0x6a47 bytes) 000165[1] 1790920136.549302 trunc from 0x76a606 to 0x69ec7f (0xcb986 bytes) 000168[1] 1790920136.704001 trunc from 0x69ec80 to 0x06a2d7 (0x6349a8 bytes) 000169[1] 1790920136.730058 mapwrite 0x164e62 thru 0x171153 (0xc2f2 bytes) 000171[0] 1790920136.834868 punch from 0x14993f to 0x149bf1 (0x2b3 bytes) 000174[1] 1790920136.897217 mapwrite 0x28c5ba thru 0x2905db (0x4022 bytes) 000177[1] 1790920137.239747 punch from 0x23a149 to 0x241987 (0x783f bytes) 000180[0] 1790920137.330642 punch from 0x107422 to 0x113d5e (0xc93d bytes) 000181[1] 1790920137.439566 trunc from 0x2905dc to 0x1a5d2e (0xea8ad bytes) 000183[1] 1790920137.520821 trunc from 0x1a5d2f to 0x23532c (0x8f5fe bytes) 000184[1] 1790920137.664202 read 0x0c0530 thru 0x0c90ee (0x8bbf bytes) 000185[1] 1790920137.771898 trunc from 0x23532d to 0x47f9ea (0x24a6be bytes) 000187[0] 1790920137.904985 trunc from 0x47f9eb to 0x64db13 (0x1ce129 bytes) 000190[1] 1790920138.008370 read 0x145025 thru 0x150763 (0xb73f bytes) 000191[1] 1790920138.047826 mapread 0x4479c8 thru 0x44da08 (0x6041 bytes) 000196[1] 1790920138.184759 read 0x5e4b7f thru 0x5e76a8 (0x2b2a bytes) 000197[0] 1790920138.192724 write 0x0ca844 thru 0x0ced11 (0x44ce bytes) 000198[1] 1790920138.322057 read 0x1cf81c thru 0x1dc99d (0xd182 bytes) 000199[1] 1790920138.369167 mapwrite 0x49f24e thru 0x4a4741 (0x54f4 bytes) 000202[0] 1790920138.689169 trunc from 0x64db14 to 0x208ce3 (0x444e30 bytes) 000203[0] 1790920138.800787 mapwrite 0x625260 thru 0x632f66 (0xdd07 bytes) 000207[1] 1790920139.182175 write 0x1ffe57 thru 0x204f28 (0x50d2 bytes) 000208[0] 1790920139.270305 trunc from 0x632f67 to 0x6de7bd (0xab857 bytes) 000209[1] 1790920139.358336 write 0x381c7f thru 0x38716b (0x54ed bytes) 000211[1] 1790920139.438384 punch from 0x469415 to 0x4780b4 (0xeca0 bytes) 000214[1] 1790920139.484289 read 0x25ddb2 thru 0x25f357 (0x15a6 bytes) 000219[1] 1790920139.573434 punch from 0x677967 to 0x681412 (0x9aac bytes) 000220[1] 1790920139.584157 mapread 0x126e52 thru 0x12aa6a (0x3c19 bytes) 000221[1] 1790920139.651223 trunc from 0x6de7be to 0x128fdc (0x5b57e1 bytes) 000222[0] 1790920139.694593 mapread 0x06e4c5 thru 0x077d64 (0x98a0 bytes) 000223[0] 1790920139.821346 mapwrite 0x699f18 thru 0x69de3c (0x3f25 bytes) 000224[1] 1790920140.073520 punch from 0x0de774 to 0x0deada (0x367 bytes) 000225[0] 1790920140.127387 trunc from 0x69de3d to 0x9ce6d5 (0x330899 bytes) 000227[1] 1790920140.241552 trunc from 0x9ce6d6 to 0x9d25eb (0x3f16 bytes) 000231[0] 1790920140.455503 punch from 0x25e3a7 to 0x26337c (0x4fd6 bytes) 000232[0] 1790920140.524670 mapwrite 0x628079 thru 0x636ce6 (0xec6e bytes) 000234[1] 1790920140.736545 mapread 0x8632ea thru 0x8673ab (0x40c2 bytes) 000235[0] 1790920140.816912 read 0x8e75c8 thru 0x8eae3e (0x3877 bytes) 000236[1] 1790920140.834917 read 0x4dadb8 thru 0x4dcfde (0x2227 bytes) 000239[1] 1790920140.927636 mapread 0x3cea12 thru 0x3da136 (0xb725 bytes) 000240[1] 1790920140.992432 mapread 0x9a8f36 thru 0x9b1d67 (0x8e32 bytes) 000242[1] 1790920141.044949 mapwrite 0x48901b thru 0x48a705 (0x16eb bytes) 000243[0] 1790920141.223377 mapwrite 0x29067c thru 0x2a0309 (0xfc8e bytes) 000244[1] 1790920141.346912 mapwrite 0x9d95e2 thru 0x9e91a5 (0xfbc4 bytes) 000245[0] 1790920141.477054 punch from 0x83c206 to 0x843210 (0x700b bytes) 000246[0] 1790920141.502271 read 0x1de880 thru 0x1e55c8 (0x6d49 bytes) 000247[1] 1790920141.525522 punch from 0x232889 to 0x23dbaa (0xb322 bytes) 000248[0] 1790920141.585114 trunc from 0x9e91a6 to 0x0923c5 (0x956de0 bytes) 000249[1] 1790920141.660989 write 0x164a15 thru 0x16eb3a (0xa126 bytes) 000250[0] 1790920141.706512 punch from 0x10b465 to 0x11283f (0x73db bytes) 000252[0] 1790920141.775394 mapread 0x0d626e thru 0x0d7172 (0xf05 bytes) 000253[0] 1790920141.836076 write 0x47985b thru 0x47df52 (0x46f8 bytes) 000257[0] 1790920141.920295 trunc from 0x47df53 to 0x5a0442 (0x1224f0 bytes) 000258[0] 1790920141.956036 read 0x1af221 thru 0x1bed22 (0xfb02 bytes) 000259[1] 1790920142.003000 write 0x7e28a6 thru 0x7eb696 (0x8df1 bytes) 000262[0] 1790920142.135494 mapwrite 0x3093c3 thru 0x30d745 (0x4383 bytes) 000263[1] 1790920142.231566 mapwrite 0x63c724 thru 0x6438b5 (0x7192 bytes) 000267[0] 1790920142.468291 read 0x449776 thru 0x4594c8 (0xfd53 bytes) 000268[1] 1790920142.492659 read 0x0a33ee thru 0x0a41c0 (0xdd3 bytes) 000269[0] 1790920142.532036 punch from 0x20364f to 0x211006 (0xd9b8 bytes) 000270[0] 1790920142.576205 punch from 0x6bdb41 to 0x6c6580 (0x8a40 bytes) 000271[0] 1790920142.653653 mapwrite 0x3736eb thru 0x3762ef (0x2c05 bytes) 000272[1] 1790920142.890245 read 0x7be116 thru 0x7cdebd (0xfda8 bytes) 000273[0] 1790920142.931337 write 0x9abc0c thru 0x9b55ae (0x99a3 bytes) 000274[0] 1790920142.960214 write 0x38b963 thru 0x39af09 (0xf5a7 bytes) 000276[1] 1790920143.030781 write 0x22760a thru 0x22bfc0 (0x49b7 bytes) 000277[1] 1790920143.102786 write 0x6ae390 thru 0x6b89ab (0xa61c bytes) 000278[0] 1790920143.130514 punch from 0x1d43af to 0x1d4b2f (0x781 bytes) 000279[0] 1790920143.176918 punch from 0x17d686 to 0x18bc3d (0xe5b8 bytes) 000281[1] 1790920143.210094 write 0x2651fc thru 0x26c805 (0x760a bytes) 000282[0] 1790920143.253060 trunc from 0x9b55af to 0x96315a (0x52454 bytes) 000285[0] 1790920143.339523 mapwrite 0x03f235 thru 0x04139c (0x2168 bytes) 000287[0] 1790920143.442133 punch from 0x8befa6 to 0x8c1d22 (0x2d7d bytes) 000288[1] 1790920143.446788 write 0x00ad2a thru 0x01a4d2 (0xf7a9 bytes) 000290[1] 1790920143.497281 mapwrite 0x897314 thru 0x8a05c7 (0x92b4 bytes) 000291[0] 1790920143.586739 punch from 0x657611 to 0x65c1cc (0x4bbc bytes) 000293[1] 1790920143.616647 mapwrite 0x07996f thru 0x079f18 (0x5aa bytes) 000294[1] 1790920143.705261 mapread 0x6bbb68 thru 0x6bceb2 (0x134b bytes) 000296[0] 1790920143.735288 read 0x902923 thru 0x91022a (0xd908 bytes) 000299[0] 1790920143.763669 mapwrite 0x6ee457 thru 0x6ee74d (0x2f7 bytes) 000301[0] 1790920143.870984 read 0x293f6d thru 0x29ba94 (0x7b28 bytes) 000303[1] 1790920143.913462 read 0x67dd3d thru 0x68173c (0x3a00 bytes) 000304[0] 1790920143.938538 punch from 0x824247 to 0x83380a (0xf5c4 bytes) 000306[1] 1790920143.964946 read 0x3b00e3 thru 0x3bbb4c (0xba6a bytes) 000307[1] 1790920144.001959 trunc from 0x96315b to 0x5fdbd6 (0x365584 bytes) 000308[1] 1790920144.106086 trunc from 0x5fdbd7 to 0x10784d (0x4f6389 bytes) 000309[0] 1790920144.175225 trunc from 0x10784e to 0x8b68a3 (0x7af056 bytes) 000314[1] 1790920144.283499 punch from 0x63caea to 0x645387 (0x889e bytes) 000318[0] 1790920144.387802 trunc from 0x8b68a4 to 0x9ce336 (0x117a93 bytes) 000321[1] 1790920144.535034 write 0x412a28 thru 0x418132 (0x570b bytes) 000322[0] 1790920144.589750 punch from 0x852ea7 to 0x854e12 (0x1f6c bytes) 000325[1] 1790920144.663816 mapwrite 0x839e82 thru 0x83b992 (0x1b11 bytes) 000327[1] 1790920145.092262 mapwrite 0x5e9310 thru 0x5e9609 (0x2fa bytes) 000328[0] 1790920145.353437 punch from 0x1cd389 to 0x1cfc14 (0x288c bytes) 000329[1] 1790920145.399080 trunc from 0x9ce337 to 0x47e6eb (0x54fc4b bytes) 000330[0] 1790920145.462799 punch from 0x18fe21 to 0x19b6f0 (0xb8d0 bytes) 000333[1] 1790920145.558770 trunc from 0x47e6ec to 0x2e649d (0x19824e bytes) 000335[1] 1790920145.614372 mapread 0x1dc7df thru 0x1dd427 (0xc49 bytes) 000337[1] 1790920145.721547 read 0x1de365 thru 0x1df8ee (0x158a bytes) 000338[0] 1790920145.821189 mapwrite 0x2012e2 thru 0x208282 (0x6fa1 bytes) 000341[0] 1790920146.113076 mapread 0x20cce2 thru 0x215b9a (0x8eb9 bytes) 000342[0] 1790920146.174331 mapwrite 0x781ee5 thru 0x78b959 (0x9a75 bytes) 000343[1] 1790920146.404178 write 0x76d49a thru 0x770479 (0x2fe0 bytes) 000344[1] 1790920146.468612 mapread 0x27a4ea thru 0x28595d (0xb474 bytes) 000345[0] 1790920146.693057 write 0x1f99c8 thru 0x1ffedf (0x6518 bytes) 000348[1] 1790920146.801184 trunc from 0x78b95a to 0x2ec3dd (0x49f57c bytes) 000350[1] 1790920146.924141 trunc from 0x2ec3de to 0x7b819f (0x4cbdc2 bytes) 000354[0] 1790920147.091345 trunc from 0x7b81a0 to 0x51a41c (0x29dd83 bytes) 000355[0] 1790920147.141350 write 0x11e710 thru 0x12a47c (0xbd6d bytes) 000358[1] 1790920147.180790 mapwrite 0x636860 thru 0x641144 (0xa8e5 bytes) 000360[0] 1790920147.396117 mapwrite 0x964fa4 thru 0x971bdd (0xcc3a bytes) 000361[0] 1790920147.622610 punch from 0x2e21b1 to 0x2e4007 (0x1e57 bytes) 000362[1] 1790920147.654559 punch from 0x33352d to 0x3384c8 (0x4f9c bytes) 000363[0] 1790920147.728875 trunc from 0x971bde to 0x599375 (0x3d8868 bytes) 000364[1] 1790920147.820665 mapread 0x137539 thru 0x13df93 (0x6a5b bytes) 000366[1] 1790920148.052445 trunc from 0x599376 to 0x87f2e3 (0x2e5f6e bytes) 000367[1] 1790920148.167029 mapwrite 0x5a1a7c thru 0x5ac3a9 (0xa92e bytes) 000368[0] 1790920148.362226 punch from 0x014df4 to 0x01d3f4 (0x8601 bytes) 000369[0] 1790920148.396057 punch from 0x48becb to 0x49a02e (0xe164 bytes) 000371[0] 1790920148.452370 read 0x1f7138 thru 0x203b4f (0xca18 bytes) 000372[0] 1790920148.503837 mapread 0x5b03a2 thru 0x5b4a83 (0x46e2 bytes) 000373[1] 1790920148.524110 trunc from 0x87f2e4 to 0x37379b (0x50bb48 bytes) 000374[0] 1790920148.601705 write 0x8d09aa thru 0x8db11e (0xa775 bytes) 000375[1] 1790920148.661142 write 0x98aeb3 thru 0x99859e (0xd6ec bytes) 000376[0] 1790920148.721189 trunc from 0x99859f to 0x0b9d9e (0x8de800 bytes) 000377[0] 1790920148.827141 mapread 0x00ed32 thru 0x01e677 (0xf946 bytes) 000379[1] 1790920148.906735 trunc from 0x0b9d9f to 0x4613be (0x3a7620 bytes) 000380[1] 1790920149.008785 write 0x10a564 thru 0x10e16d (0x3c0a bytes) 000381[0] 1790920149.021749 mapread 0x4402ff thru 0x44e05a (0xdd5c bytes) 000382[1] 1790920149.189261 read 0x3b489b thru 0x3c05b4 (0xbd1a bytes) 000384[1] 1790920149.254492 read 0x25201b thru 0x25e883 (0xc869 bytes) 000387[1] 1790920149.326504 punch from 0x18fe00 to 0x19b36e (0xb56f bytes) 000388[0] 1790920149.350997 read 0x1386f4 thru 0x141375 (0x8c82 bytes) 000389[1] 1790920149.382712 write 0x4cf284 thru 0x4d6b8c (0x7909 bytes) 000390[0] 1790920149.400223 punch from 0x31143d to 0x31f5fa (0xe1be bytes) 000391[0] 1790920149.479353 write 0x5101f8 thru 0x5163c8 (0x61d1 bytes) 000392[0] 1790920149.572241 write 0x78012e thru 0x78e7c1 (0xe694 bytes) 000393[0] 1790920149.624370 mapread 0x10a97b thru 0x10c118 (0x179e bytes) 000394[1] 1790920149.690314 write 0x6e73d7 thru 0x6f3645 (0xc26f bytes) 000395[1] 1790920149.758231 trunc from 0x78e7c2 to 0x9ad680 (0x21eebf bytes) 000396[1] 1790920149.837369 write 0x8dde3c thru 0x8de3ff (0x5c4 bytes) 000397[1] 1790920149.853984 mapread 0x3fb0b9 thru 0x406fd4 (0xbf1c bytes) 000400[0] 1790920149.999962 write 0x6dc250 thru 0x6e699a (0xa74b bytes) 000401[0] 1790920150.075117 trunc from 0x9ad681 to 0x1e04f8 (0x7cd188 bytes) 000402[1] 1790920150.145199 trunc from 0x1e04f9 to 0x35334c (0x172e54 bytes) 000405[0] 1790920150.293748 punch from 0x0b3cf8 to 0x0bac67 (0x6f70 bytes) 000406[1] 1790920150.346020 trunc from 0x35334d to 0x717e4f (0x3c4b03 bytes) 000407[1] 1790920150.437289 mapread 0x5fb123 thru 0x608929 (0xd807 bytes) 000408[0] 1790920150.502265 write 0x2978fd thru 0x29ba2f (0x4133 bytes) 000409[1] 1790920150.533353 punch from 0x607cec to 0x60d9b7 (0x5ccc bytes) 000411[0] 1790920150.570542 punch from 0x6dac75 to 0x6e4758 (0x9ae4 bytes) 000412[1] 1790920150.604848 punch from 0x65cb66 to 0x663ef1 (0x738c bytes) 000414[1] 1790920150.645262 read 0x284c1b thru 0x285691 (0xa77 bytes) 000415[0] 1790920150.666760 write 0x2c3128 thru 0x2c746d (0x4346 bytes) 000420[0] 1790920150.749983 write 0x5ac48d thru 0x5b5ba5 (0x9719 bytes) 000421[1] 1790920150.778970 read 0x13025d thru 0x13ffad (0xfd51 bytes) 000423[0] 1790920150.809385 trunc from 0x717e50 to 0x1baab2 (0x55d39d bytes) 000425[0] 1790920150.865488 mapwrite 0x40c2dd thru 0x40d1b7 (0xedb bytes) 000426[0] 1790920150.938703 trunc from 0x40d1b8 to 0x0a0470 (0x36cd47 bytes) 000428[0] 1790920150.977697 trunc from 0x0a0471 to 0x0fa067 (0x59bf7 bytes) 000430[0] 1790920151.031931 mapread 0x07bc09 thru 0x0869a7 (0xad9f bytes) 000433[0] 1790920151.115590 mapwrite 0x4d51c6 thru 0x4d6111 (0xf4c bytes) 000434[0] 1790920151.350359 mapwrite 0x479b7f thru 0x4808c6 (0x6d48 bytes) 000436[1] 1790920151.522031 mapread 0x4c5f05 thru 0x4d365c (0xd758 bytes) 000438[1] 1790920151.602793 mapwrite 0x0ecc8b thru 0x0f7ddf (0xb155 bytes) 000439[1] 1790920151.760726 trunc from 0x4d6112 to 0x1032ba (0x3d2e57 bytes) 000440[1] 1790920151.842218 trunc from 0x1032bb to 0x6c4628 (0x5c136e bytes) 000441[0] 1790920151.886889 write 0x94d99c thru 0x94da6f (0xd4 bytes) 000443[1] 1790920151.959467 write 0x684705 thru 0x688e76 (0x4772 bytes) 000445[1] 1790920152.076316 write 0x2539fc thru 0x261212 (0xd817 bytes) 000446[0] 1790920152.158937 trunc from 0x94da70 to 0x4be272 (0x48f7fd bytes) 000447[1] 1790920152.288326 write 0x91a212 thru 0x925069 (0xae58 bytes) 000448[1] 1790920152.375296 punch from 0x5c3a58 to 0x5c79bf (0x3f68 bytes) 000449[1] 1790920152.415728 read 0x67a0cf thru 0x68270b (0x863d bytes) 000453[0] 1790920152.495137 punch from 0x324bfb to 0x32965a (0x4a60 bytes) 000454[1] 1790920152.546039 write 0x414f48 thru 0x4230a0 (0xe159 bytes) 000456[1] 1790920152.598136 punch from 0x13c0fc to 0x1420b9 (0x5fbe bytes) 000458[1] 1790920152.637113 punch from 0x6ea396 to 0x6efac5 (0x5730 bytes) 000461[1] 1790920152.701945 write 0x3f25d5 thru 0x3fede3 (0xc80f bytes) 000463[1] 1790920152.761191 read 0x230ca6 thru 0x230ea2 (0x1fd bytes) 000464[0] 1790920152.808651 punch from 0x5e849d to 0x5f35ff (0xb163 bytes) 000466[0] 1790920152.894263 mapwrite 0x81015d thru 0x8183d3 (0x8277 bytes) 000469[0] 1790920153.298744 trunc from 0x92506a to 0x1e9696 (0x73b9d3 bytes) 000472[0] 1790920153.380321 write 0x92ea2b thru 0x930b12 (0x20e8 bytes) 000473[0] 1790920153.388736 write 0x7b8bbc thru 0x7c11e2 (0x8627 bytes) 000474[1] 1790920153.397098 mapwrite 0x750518 thru 0x758937 (0x8420 bytes) 000476[1] 1790920153.545205 mapread 0x5d2f8e thru 0x5db063 (0x80d6 bytes) 000477[1] 1790920153.585171 write 0x025e50 thru 0x0309da (0xab8b bytes) 000478[1] 1790920153.658118 mapwrite 0x77bc53 thru 0x7826c8 (0x6a76 bytes) 000482[0] 1790920153.948322 punch from 0x127746 to 0x1346fd (0xcfb8 bytes) 000483[1] 1790920154.025233 trunc from 0x930b13 to 0x32b9e5 (0x60512d bytes) 000485[0] 1790920154.120589 punch from 0x06aae1 to 0x07816a (0xd68a bytes) 000486[0] 1790920154.153007 trunc from 0x32b9e6 to 0x7d51bb (0x4a97d6 bytes) 000489[0] 1790920154.205237 write 0x8c119a thru 0x8c4daf (0x3c16 bytes) 000491[1] 1790920154.228673 write 0x51ef4c thru 0x5204ef (0x15a4 bytes) 000492[1] 1790920154.266728 mapwrite 0x5c876a thru 0x5d7ccf (0xf566 bytes) 000493[0] 1790920154.386034 mapwrite 0x485c01 thru 0x489959 (0x3d59 bytes) 000494[1] 1790920154.480139 trunc from 0x8c4db0 to 0x14d052 (0x777d5d bytes) 000495[0] 1790920154.539958 mapread 0x142e9e thru 0x1457d2 (0x2935 bytes) 000496[0] 1790920154.635365 write 0x04697c thru 0x048327 (0x19ac bytes) 000497[1] 1790920154.696242 mapread 0x06bbda thru 0x078f81 (0xd3a8 bytes) 000498[1] 1790920154.769172 read 0x03743b thru 0x0426aa (0xb270 bytes) 000499[1] 1790920154.815374 punch from 0x0a6213 to 0x0b1286 (0xb074 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 155 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole 000004[1] 1790920156.018742 write 0x1d5000 thru 0x1dde86 (0x8e87 bytes) 000005[0] 1790920156.021828 mapwrite 0x457000 thru 0x45b072 (0x4073 bytes) 000006[0] 1790920156.193895 punch from 0x129987 to 0x12a66a (0xce4 bytes) truncating to largest ever: 0x5e105b 000008[0] 1790920156.243064 trunc from 0x45b073 to 0x5e105a (0x185fe8 bytes) 000010[0] 1790920156.305309 write_OD 0x1a4000 thru 0x1aefff (0xb000 bytes) 000012[1] 1790920156.372381 read 0x3d2000 thru 0x3d97d4 (0x77d5 bytes) 000014[1] 1790920156.384781 trunc from 0x5e105b to 0x4c469c (0x11c9be bytes) 000015[0] 1790920156.484520 write_OD 0x1ba000 thru 0x1c5fff (0xc000 bytes) 000016[1] 1790920156.603467 read 0x26e000 thru 0x271fab (0x3fac bytes) 000020[0] 1790920156.640362 trunc from 0x4c469d to 0x15b201 (0x36949b bytes) 000024[0] 1790920156.734866 punch from 0x0cb4b4 to 0x0d0b52 (0x569f bytes) 000026[0] 1790920156.770120 punch from 0x053fcc to 0x057694 (0x36c9 bytes) 000027[0] 1790920156.781089 mapwrite 0x872000 thru 0x878cc6 (0x6cc7 bytes) truncating to largest ever: 0x7c2f8a 000028[1] 1790920156.922869 trunc from 0x878cc7 to 0x7c2f89 (0xb5d3d bytes) 000029[0] 1790920156.999156 write_OD 0x4b1000 thru 0x4b5fff (0x5000 bytes) 000030[1] 1790920157.199147 write 0x2a9000 thru 0x2ae959 (0x595a bytes) 000032[1] 1790920157.282940 punch from 0x16261f to 0x16e2d2 (0xbcb4 bytes) 000033[1] 1790920157.328888 punch from 0x02add8 to 0x034192 (0x93bb bytes) 000034[0] 1790920157.334157 write_OD 0x586000 thru 0x58efff (0x9000 bytes) 000035[1] 1790920157.515202 read 0x292000 thru 0x29ee26 (0xce27 bytes) 000036[0] 1790920157.547886 write_OD 0x93f000 thru 0x943fff (0x5000 bytes) 000038[1] 1790920157.667635 trunc from 0x944000 to 0x2438db (0x700724 bytes) 000039[0] 1790920157.717307 punch from 0x1906ea to 0x1951ab (0x4ac2 bytes) 000041[0] 1790920157.751697 write_OD 0x179000 thru 0x17efff (0x6000 bytes) 000042[1] 1790920157.826870 trunc from 0x2438dc to 0x019177 (0x22a764 bytes) 000044[0] 1790920157.920248 read_OD 0x009000 thru 0x00cfff (0x4000 bytes) 000046[0] 1790920157.993229 trunc from 0x019178 to 0x522461 (0x5092ea bytes) 000048[1] 1790920158.080889 mapwrite 0x577000 thru 0x57a777 (0x3778 bytes) 000051[1] 1790920158.340072 punch from 0x215858 to 0x21681b (0xfc4 bytes) 000052[0] 1790920158.367477 mapread 0x427000 thru 0x42e01e (0x701f bytes) 000053[0] 1790920158.471194 punch from 0x551bf4 to 0x561b5c (0xff69 bytes) 000054[0] 1790920158.562523 mapread 0x2d0000 thru 0x2dc40c (0xc40d bytes) 000056[0] 1790920158.639491 write 0x29c000 thru 0x29d71c (0x171d bytes) 000057[0] 1790920158.659202 mapread 0x47f000 thru 0x48a4b4 (0xb4b5 bytes) 000058[0] 1790920158.755126 mapwrite 0x126000 thru 0x12c2e3 (0x62e4 bytes) 000060[1] 1790920158.974919 punch from 0x0caa2b to 0x0cb521 (0xaf7 bytes) 000062[0] 1790920159.014113 mapwrite 0x12c000 thru 0x1371bb (0xb1bc bytes) 000063[1] 1790920159.323828 mapread 0x284000 thru 0x28a0cb (0x60cc bytes) 000064[1] 1790920159.452703 read 0x579000 thru 0x57a573 (0x1574 bytes) 000066[0] 1790920159.466260 mapread 0x29b000 thru 0x29b990 (0x991 bytes) 000067[0] 1790920159.487153 mapwrite 0x480000 thru 0x483f77 (0x3f78 bytes) 000068[1] 1790920159.679816 trunc from 0x57a778 to 0x24e5c8 (0x32c1af bytes) 000069[1] 1790920159.750269 punch from 0x243a85 to 0x24e5c8 (0xab44 bytes) 000070[1] 1790920159.761245 write 0x205000 thru 0x210588 (0xb589 bytes) 000071[0] 1790920159.807317 trunc from 0x24e5c9 to 0x65e064 (0x40fa9c bytes) 000072[1] 1790920159.909171 read 0x1b8000 thru 0x1b9670 (0x1671 bytes) 000073[1] 1790920159.946376 trunc from 0x65e065 to 0x035bb0 (0x6284b4 bytes) 000074[0] 1790920160.004298 trunc from 0x035bb1 to 0x56d31f (0x53776f bytes) 000076[0] 1790920160.071577 mapwrite 0x19b000 thru 0x19c45f (0x1460 bytes) 000077[1] 1790920160.138710 read 0x44c000 thru 0x44f78c (0x378d bytes) 000078[0] 1790920160.152049 read 0x4aa000 thru 0x4b632a (0xc32b bytes) 000079[0] 1790920160.159194 read 0x55e000 thru 0x56c59e (0xe59f bytes) 000080[0] 1790920160.167359 punch from 0x02e90a to 0x0354f4 (0x6beb bytes) 000082[0] 1790920160.216185 write 0x5fc000 thru 0x607629 (0xb62a bytes) 000084[1] 1790920160.229079 mapwrite 0x845000 thru 0x853362 (0xe363 bytes) 000089[0] 1790920160.454946 punch from 0x2bee5b to 0x2c3570 (0x4716 bytes) 000090[1] 1790920160.513036 mapwrite 0x1c4000 thru 0x1d3402 (0xf403 bytes) 000093[0] 1790920160.727808 trunc from 0x853363 to 0x1fd161 (0x656201 bytes) 000094[0] 1790920160.793010 trunc from 0x1fd162 to 0x4fb3ce (0x2fe26d bytes) 000096[0] 1790920160.844577 read 0x441000 thru 0x44be08 (0xae09 bytes) 000097[1] 1790920160.883195 mapread 0x09b000 thru 0x09f2a1 (0x42a2 bytes) 000098[0] 1790920160.985338 mapread 0x4cb000 thru 0x4d413c (0x913d bytes) 000099[0] 1790920161.026664 write 0x6de000 thru 0x6ecbaa (0xebab bytes) 000101[0] 1790920161.135690 read 0x27e000 thru 0x28a80e (0xc80f bytes) truncating to largest ever: 0x9542a1 000104[1] 1790920161.229284 trunc from 0x6ecbab to 0x9542a0 (0x2676f6 bytes) 000110[0] 1790920161.515541 mapwrite 0x198000 thru 0x19c7a0 (0x47a1 bytes) 000112[1] 1790920161.751117 trunc from 0x9542a1 to 0x390c49 (0x5c3657 bytes) 000114[0] 1790920161.797306 punch from 0x04f3ab to 0x05390d (0x4563 bytes) 000118[0] 1790920161.887983 punch from 0x2976e9 to 0x2978f6 (0x20e bytes) 000120[1] 1790920161.895069 read_OD 0x0ab000 thru 0x0b4fff (0xa000 bytes) 000121[1] 1790920161.933734 write_OD 0x0db000 thru 0x0e5fff (0xb000 bytes) 000122[1] 1790920162.011563 mapwrite 0x8e5000 thru 0x8e9847 (0x4848 bytes) 000123[1] 1790920162.249888 trunc from 0x8e9848 to 0x29afe4 (0x64e863 bytes) 000126[1] 1790920162.303304 trunc from 0x29afe5 to 0x4baaa9 (0x21fac5 bytes) 000127[0] 1790920162.396189 punch from 0x00134b to 0x0103e2 (0xf098 bytes) 000128[1] 1790920162.417638 mapwrite 0x8cc000 thru 0x8dba8d (0xfa8e bytes) 000129[0] 1790920162.608982 write 0x878000 thru 0x88263c (0xa63d bytes) 000132[1] 1790920162.736046 mapread 0x1e7000 thru 0x1f18a9 (0xa8aa bytes) 000133[1] 1790920162.785488 mapwrite 0x141000 thru 0x147b49 (0x6b4a bytes) 000135[1] 1790920162.939068 trunc from 0x8dba8e to 0x2ed3b3 (0x5ee6da bytes) 000136[1] 1790920162.974960 mapread 0x2b9000 thru 0x2c365d (0xa65e bytes) truncating to largest ever: 0x95b94e 000138[1] 1790920163.079045 trunc from 0x2ed3b4 to 0x95b94d (0x66e59a bytes) 000139[1] 1790920163.149301 read 0x118000 thru 0x127f15 (0xff16 bytes) 000140[0] 1790920163.255933 read 0x8c9000 thru 0x8d8a32 (0xfa33 bytes) 000141[1] 1790920163.377683 trunc from 0x95b94e to 0x85cb6b (0xfede2 bytes) 000142[0] 1790920163.476863 mapwrite 0x16e000 thru 0x176bbc (0x8bbd bytes) 000143[0] 1790920163.649378 trunc from 0x85cb6c to 0x50d2b9 (0x34f8b2 bytes) 000144[0] 1790920163.715535 mapread 0x0ff000 thru 0x0ff46d (0x46e bytes) 000145[1] 1790920163.757023 mapread 0x12a000 thru 0x135929 (0xb92a bytes) 000147[1] 1790920163.825257 trunc from 0x50d2ba to 0x6630bc (0x155e03 bytes) 000148[1] 1790920163.856310 trunc from 0x6630bd to 0x545330 (0x11dd8c bytes) 000149[0] 1790920163.873777 trunc from 0x545331 to 0x1ef7c9 (0x355b67 bytes) 000150[1] 1790920163.948096 mapwrite 0x00d000 thru 0x0104b1 (0x34b2 bytes) 000154[1] 1790920164.262728 mapread 0x0e0000 thru 0x0e7214 (0x7215 bytes) 000156[0] 1790920164.326175 punch from 0x14a561 to 0x153319 (0x8db9 bytes) 000157[0] 1790920164.391050 write 0x5e5000 thru 0x5ecb12 (0x7b13 bytes) 000158[0] 1790920164.393104 mapread 0x5b4000 thru 0x5ba559 (0x655a bytes) 000159[1] 1790920164.423823 mapwrite 0x899000 thru 0x89a7f9 (0x17fa bytes) 000161[0] 1790920164.538342 mapread 0x69d000 thru 0x6a8ebe (0xbebf bytes) 000162[1] 1790920164.570569 write 0x156000 thru 0x160944 (0xa945 bytes) 000166[1] 1790920164.579764 read 0x13a000 thru 0x148100 (0xe101 bytes) 000167[1] 1790920164.611228 mapread 0x057000 thru 0x05c274 (0x5275 bytes) 000172[0] 1790920164.688587 mapwrite 0x360000 thru 0x3606e2 (0x6e3 bytes) 000173[1] 1790920164.875059 trunc from 0x89a7fa to 0x34ba83 (0x54ed76 bytes) 000175[0] 1790920165.029772 read 0x12d000 thru 0x1317c0 (0x47c1 bytes) 000176[0] 1790920165.213753 trunc from 0x34ba84 to 0x508c34 (0x1bd1b1 bytes) 000177[1] 1790920165.270376 read 0x106000 thru 0x10b2a2 (0x52a3 bytes) 000178[0] 1790920165.372901 write 0x318000 thru 0x327f5b (0xff5c bytes) 000179[1] 1790920165.485204 read 0x3d3000 thru 0x3d6ddb (0x3ddc bytes) 000183[1] 1790920165.582843 punch from 0x4d8371 to 0x4e646f (0xe0ff bytes) 000185[0] 1790920165.715283 mapwrite 0x13a000 thru 0x140634 (0x6635 bytes) 000186[0] 1790920166.042164 mapwrite 0x514000 thru 0x51652f (0x2530 bytes) 000189[0] 1790920166.328891 punch from 0x392630 to 0x39728e (0x4c5f bytes) 000190[1] 1790920166.358208 mapwrite 0x45d000 thru 0x45d83b (0x83c bytes) 000191[1] 1790920166.453025 trunc from 0x516530 to 0x8b636d (0x39fe3e bytes) 000192[1] 1790920166.514072 read 0x3c0000 thru 0x3cf09a (0xf09b bytes) 000194[0] 1790920166.537299 mapwrite 0x09e000 thru 0x09e1cb (0x1cc bytes) 000197[1] 1790920166.695608 read 0x4a8000 thru 0x4ab601 (0x3602 bytes) 000198[1] 1790920166.726113 punch from 0x6ece42 to 0x6f2937 (0x5af6 bytes) 000199[0] 1790920166.755158 read 0x76f000 thru 0x776ff4 (0x7ff5 bytes) 000200[0] 1790920166.776854 read 0x59f000 thru 0x5a9b2c (0xab2d bytes) 000201[0] 1790920166.788546 write 0x750000 thru 0x758768 (0x8769 bytes) 000202[0] 1790920166.825725 trunc from 0x8b636e to 0x2e83c0 (0x5cdfad bytes) 000205[1] 1790920166.915588 write 0x552000 thru 0x55645b (0x445c bytes) 000208[0] 1790920167.008066 write 0x472000 thru 0x47851c (0x651d bytes) 000209[0] 1790920167.062890 punch from 0x1c3178 to 0x1c4d7f (0x1c08 bytes) 000210[0] 1790920167.107946 trunc from 0x55645c to 0x0368c8 (0x51fb93 bytes) 000211[0] 1790920167.143068 read 0x02f000 thru 0x0365f0 (0x75f1 bytes) 000213[1] 1790920167.206081 write 0x998000 thru 0x9a357b (0xb57c bytes) 000214[0] 1790920167.221134 trunc from 0x9a357c to 0x26a4f9 (0x739082 bytes) 000215[0] 1790920167.279214 punch from 0x177f3e to 0x17807a (0x13d bytes) 000216[1] 1790920167.287207 punch from 0x01a419 to 0x024164 (0x9d4c bytes) 000217[0] 1790920167.354144 mapread 0x17a000 thru 0x186e30 (0xce31 bytes) 000218[1] 1790920167.509719 punch from 0x0f33dd to 0x0ff572 (0xc196 bytes) 000219[0] 1790920167.605989 punch from 0x0c69df to 0x0c71da (0x7fc bytes) 000221[0] 1790920167.708831 mapread 0x1a5000 thru 0x1a640d (0x140e bytes) 000222[1] 1790920167.735611 punch from 0x2201dd to 0x2277a7 (0x75cb bytes) 000223[1] 1790920167.780316 write 0x59d000 thru 0x5aca07 (0xfa08 bytes) 000224[1] 1790920167.805015 mapwrite 0x33e000 thru 0x349e38 (0xbe39 bytes) 000225[1] 1790920168.017264 write 0x044000 thru 0x05116e (0xd16f bytes) 000226[1] 1790920168.035745 punch from 0x4f5664 to 0x4fa4d4 (0x4e71 bytes) 000228[1] 1790920168.128689 punch from 0x4d7d60 to 0x4da370 (0x2611 bytes) 000229[1] 1790920168.150895 punch from 0x59ecb7 to 0x59efa7 (0x2f1 bytes) 000230[0] 1790920168.155319 read 0x344000 thru 0x350993 (0xc994 bytes) 000231[0] 1790920168.199659 write 0x1d8000 thru 0x1e35e1 (0xb5e2 bytes) 000233[1] 1790920168.236013 read 0x3ed000 thru 0x3f2668 (0x5669 bytes) 000234[1] 1790920168.295112 mapwrite 0x607000 thru 0x60cb2d (0x5b2e bytes) 000235[1] 1790920168.551624 mapread 0x4bf000 thru 0x4c788b (0x888c bytes) 000236[0] 1790920168.627859 read 0x052000 thru 0x05912e (0x712f bytes) 000240[0] 1790920168.713621 mapread 0x543000 thru 0x54bc47 (0x8c48 bytes) 000241[0] 1790920168.756218 mapwrite 0x7e7000 thru 0x7f00d6 (0x90d7 bytes) 000246[0] 1790920169.040464 write 0x2e0000 thru 0x2e1d88 (0x1d89 bytes) 000247[0] 1790920169.048384 mapwrite 0x8a4000 thru 0x8ae5c3 (0xa5c4 bytes) 000248[1] 1790920169.255119 punch from 0x536996 to 0x53a36f (0x39da bytes) 000249[0] 1790920169.292210 write 0x7a5000 thru 0x7aa0d9 (0x50da bytes) 000250[0] 1790920169.345330 read 0x0d0000 thru 0x0d261d (0x261e bytes) 000252[1] 1790920169.402981 mapwrite 0x429000 thru 0x431f13 (0x8f14 bytes) 000253[1] 1790920169.662854 trunc from 0x8ae5c4 to 0x3d7f64 (0x4d665f bytes) 000257[0] 1790920169.845117 punch from 0x0e462d to 0x0f2f7b (0xe94f bytes) 000258[0] 1790920169.938964 trunc from 0x3d7f65 to 0x268be9 (0x16f37b bytes) 000260[1] 1790920170.025993 punch from 0x05c837 to 0x06a826 (0xdff0 bytes) 000261[0] 1790920170.086992 mapwrite 0x673000 thru 0x67958a (0x658b bytes) 000262[0] 1790920170.342585 punch from 0x2372d2 to 0x242119 (0xae48 bytes) 000263[0] 1790920170.364506 mapread 0x5aa000 thru 0x5ae05d (0x405e bytes) 000264[0] 1790920170.469233 mapread 0x5ee000 thru 0x5f021a (0x221b bytes) 000268[1] 1790920170.559508 trunc from 0x67958b to 0x75d0f7 (0xe3b6d bytes) 000269[1] 1790920170.635967 punch from 0x213395 to 0x214107 (0xd73 bytes) 000270[0] 1790920170.658982 trunc from 0x75d0f8 to 0x330435 (0x42ccc2 bytes) 000274[1] 1790920170.724334 write 0x4ee000 thru 0x4f695d (0x895e bytes) 000276[0] 1790920170.778749 read 0x282000 thru 0x282e6c (0xe6d bytes) 000277[1] 1790920170.817766 read 0x25c000 thru 0x26399c (0x799d bytes) 000278[0] 1790920170.825297 punch from 0x25e1ed to 0x263a17 (0x582b bytes) 000280[0] 1790920170.855615 trunc from 0x4f695e to 0x246084 (0x2b08d9 bytes) 000282[0] 1790920170.903199 punch from 0x09a32e to 0x09f885 (0x5558 bytes) 000283[1] 1790920170.908030 punch from 0x00d7e5 to 0x01cdb0 (0xf5cc bytes) 000285[0] 1790920170.920634 write 0x8cd000 thru 0x8dc692 (0xf693 bytes) 000287[1] 1790920170.946851 mapread 0x1ea000 thru 0x1ee347 (0x4348 bytes) 000289[1] 1790920171.001109 punch from 0x37be10 to 0x37ffe0 (0x41d1 bytes) 000292[0] 1790920171.036089 trunc from 0x8dc693 to 0x3e8d51 (0x4f3941 bytes) 000293[0] 1790920171.114045 write 0x57e000 thru 0x5842ad (0x62ae bytes) 000294[1] 1790920171.145341 trunc from 0x5842ae to 0x7627b6 (0x1de509 bytes) 000295[0] 1790920171.276363 punch from 0x155495 to 0x15d6d3 (0x823f bytes) 000296[1] 1790920171.305765 write 0x406000 thru 0x41453f (0xe540 bytes) 000297[0] 1790920171.341802 mapwrite 0x702000 thru 0x70e25e (0xc25f bytes) 000298[1] 1790920171.578336 mapwrite 0x58a000 thru 0x5908bb (0x68bc bytes) 000299[0] 1790920171.846320 mapread 0x62b000 thru 0x63826a (0xd26b bytes) 000300[1] 1790920171.892167 trunc from 0x7627b7 to 0x3ea265 (0x378551 bytes) 000303[1] 1790920171.953547 trunc from 0x3ea266 to 0x2005f6 (0x1e9c6f bytes) 000305[0] 1790920172.000316 trunc from 0x2005f7 to 0x48b773 (0x28b17d bytes) 000306[0] 1790920172.037577 trunc from 0x48b774 to 0x38b24b (0x100528 bytes) 000308[0] 1790920172.069973 write 0x29c000 thru 0x29fd79 (0x3d7a bytes) 000309[1] 1790920172.094821 trunc from 0x38b24c to 0x4c05c9 (0x13537e bytes) 000310[0] 1790920172.143271 read 0x432000 thru 0x433e48 (0x1e49 bytes) 000311[1] 1790920172.294962 read 0x02d000 thru 0x02e6ff (0x1700 bytes) 000315[1] 1790920172.318168 punch from 0x28614b to 0x295028 (0xeede bytes) 000316[1] 1790920172.372676 write 0x731000 thru 0x740b66 (0xfb67 bytes) 000317[1] 1790920172.376961 mapwrite 0x3dd000 thru 0x3e6f51 (0x9f52 bytes) 000318[1] 1790920172.592057 write 0x9d6000 thru 0x9e5a43 (0xfa44 bytes) 000319[1] 1790920172.610217 read 0x043000 thru 0x048b43 (0x5b44 bytes) 000321[0] 1790920172.634491 punch from 0x188711 to 0x18a237 (0x1b27 bytes) 000322[1] 1790920172.666067 read 0x41d000 thru 0x41fa75 (0x2a76 bytes) 000323[1] 1790920172.703684 read 0x5be000 thru 0x5cc76b (0xe76c bytes) 000324[1] 1790920172.717815 write 0x73b000 thru 0x743b39 (0x8b3a bytes) 000325[0] 1790920172.747330 mapwrite 0x710000 thru 0x71a117 (0xa118 bytes) 000327[0] 1790920173.121087 mapread 0x4d0000 thru 0x4d3e9f (0x3ea0 bytes) 000329[1] 1790920173.155259 mapwrite 0x827000 thru 0x827716 (0x717 bytes) 000333[0] 1790920173.343935 write 0x3a7000 thru 0x3ac1c9 (0x51ca bytes) 000334[1] 1790920173.442820 mapwrite 0x895000 thru 0x8a077f (0xb780 bytes) 000336[0] 1790920173.726801 read 0x925000 thru 0x92ca7e (0x7a7f bytes) 000337[1] 1790920173.789340 trunc from 0x9e5a44 to 0x8f9d69 (0xebcda bytes) 000338[0] 1790920173.908541 punch from 0x50ee63 to 0x51ea47 (0xfbe5 bytes) 000340[1] 1790920173.948339 punch from 0x18b1cf to 0x197411 (0xc243 bytes) 000343[0] 1790920174.039950 read 0x7d5000 thru 0x7deb45 (0x9b46 bytes) 000344[0] 1790920174.058473 read 0x0c2000 thru 0x0c8714 (0x6715 bytes) 000345[1] 1790920174.072964 mapread 0x6e7000 thru 0x6f4210 (0xd211 bytes) 000346[1] 1790920174.123223 read 0x307000 thru 0x30bb04 (0x4b05 bytes) 000347[1] 1790920174.131494 mapread 0x5b2000 thru 0x5b46a1 (0x26a2 bytes) 000351[0] 1790920174.170285 mapread 0x067000 thru 0x070e4a (0x9e4b bytes) 000354[0] 1790920174.195738 mapwrite 0x0a0000 thru 0x0afaae (0xfaaf bytes) 000356[0] 1790920174.334182 mapwrite 0x97d000 thru 0x9823b9 (0x53ba bytes) 000357[1] 1790920174.465889 mapwrite 0x25a000 thru 0x25ba90 (0x1a91 bytes) 000358[0] 1790920174.708520 punch from 0x416284 to 0x4259bc (0xf739 bytes) 000359[0] 1790920174.736372 punch from 0x456f1a to 0x466949 (0xfa30 bytes) 000360[1] 1790920174.753296 mapwrite 0x0e7000 thru 0x0ef3ad (0x83ae bytes) 000361[1] 1790920175.061790 read 0x89b000 thru 0x8a3709 (0x870a bytes) 000366[1] 1790920175.182843 punch from 0x36c219 to 0x374272 (0x805a bytes) 000367[1] 1790920175.206228 mapread 0x5ab000 thru 0x5b056a (0x556b bytes) 000368[1] 1790920175.274215 punch from 0x1c4d45 to 0x1d2cd7 (0xdf93 bytes) 000373[1] 1790920175.442264 mapread 0x454000 thru 0x45e18d (0xa18e bytes) 000374[0] 1790920175.488768 trunc from 0x9823ba to 0x92a687 (0x57d32 bytes) 000376[1] 1790920175.581371 punch from 0x0f9eb3 to 0x0fdb52 (0x3ca0 bytes) 000377[1] 1790920175.618045 mapread 0x32b000 thru 0x3375c2 (0xc5c3 bytes) 000378[0] 1790920175.674132 write_OD 0x0a3000 thru 0x0a6fff (0x4000 bytes) 000379[0] 1790920175.730854 mapread 0x7e6000 thru 0x7ecf51 (0x6f52 bytes) 000380[1] 1790920175.778195 write 0x16f000 thru 0x170879 (0x187a bytes) 000381[1] 1790920175.812157 mapread 0x07a000 thru 0x085824 (0xb825 bytes) 000383[1] 1790920175.913303 trunc from 0x92a688 to 0x46f0ff (0x4bb588 bytes) 000386[1] 1790920176.139215 trunc from 0x46f100 to 0x14dab6 (0x321649 bytes) 000387[1] 1790920176.211257 punch from 0x0e3ea7 to 0x0ebac8 (0x7c22 bytes) 000389[1] 1790920176.225132 punch from 0x008b88 to 0x00d9f3 (0x4e6c bytes) 000392[1] 1790920176.349736 write 0x7ca000 thru 0x7cde24 (0x3e25 bytes) 000394[0] 1790920176.362494 trunc from 0x7cde25 to 0x511379 (0x2bcaab bytes) 000396[0] 1790920176.504631 punch from 0x4e4d5d to 0x4e6997 (0x1c3b bytes) 000398[1] 1790920176.585009 trunc from 0x51137a to 0x82ff06 (0x31eb8d bytes) 000399[1] 1790920176.665914 write 0x8fe000 thru 0x908f6d (0xaf6e bytes) 000401[1] 1790920176.719099 mapread 0x4b2000 thru 0x4b63fe (0x43ff bytes) 000402[0] 1790920176.807815 mapwrite 0x8b0000 thru 0x8b0967 (0x968 bytes) 000403[1] 1790920177.018153 mapread 0x61c000 thru 0x629b34 (0xdb35 bytes) 000404[1] 1790920177.133711 read 0x350000 thru 0x35de86 (0xde87 bytes) 000406[1] 1790920177.149786 write 0x27b000 thru 0x287bdc (0xcbdd bytes) 000407[1] 1790920177.191292 read 0x138000 thru 0x145241 (0xd242 bytes) 000408[0] 1790920177.214707 mapwrite 0x7ed000 thru 0x7ed7fa (0x7fb bytes) 000409[1] 1790920177.466549 punch from 0x851bba to 0x858808 (0x6c4f bytes) 000412[1] 1790920177.531537 read 0x54a000 thru 0x553522 (0x9523 bytes) 000415[0] 1790920177.630474 write_OD 0x1f3000 thru 0x1fbfff (0x9000 bytes) 000416[1] 1790920177.753519 mapread 0x61f000 thru 0x62dbf3 (0xebf4 bytes) 000420[1] 1790920177.820953 punch from 0x30c79c to 0x31042f (0x3c94 bytes) 000421[1] 1790920177.898039 trunc from 0x908f6e to 0x39853f (0x570a2e bytes) 000423[1] 1790920178.010335 punch from 0x31c958 to 0x32029f (0x3948 bytes) 000425[1] 1790920178.055194 mapread 0x0f3000 thru 0x0f6ffe (0x3fff bytes) 000426[1] 1790920178.131074 trunc from 0x398540 to 0x5b9dec (0x2218ad bytes) 000427[1] 1790920178.187855 trunc from 0x5b9ded to 0x4423af (0x177a3d bytes) 000428[0] 1790920178.220545 mapwrite 0x527000 thru 0x535e61 (0xee62 bytes) 000429[1] 1790920178.420041 mapwrite 0x0bc000 thru 0x0c6cfa (0xacfb bytes) 000430[0] 1790920178.682875 trunc from 0x535e62 to 0x10ea17 (0x42744a bytes) 000432[0] 1790920178.801784 write_OD 0x81b000 thru 0x827fff (0xd000 bytes) 000434[1] 1790920178.948542 read 0x0d9000 thru 0x0d91d2 (0x1d3 bytes) 000436[1] 1790920178.957826 mapread 0x6d1000 thru 0x6dd0d4 (0xc0d5 bytes) 000437[1] 1790920179.006968 punch from 0x06261a to 0x06f86f (0xd256 bytes) 000438[0] 1790920179.112674 mapwrite 0x60d000 thru 0x610287 (0x3288 bytes) 000439[0] 1790920179.386775 write_OD 0x694000 thru 0x699fff (0x6000 bytes) 000440[0] 1790920179.554239 write_OD 0x0b0000 thru 0x0b0fff (0x1000 bytes) 000441[1] 1790920179.594272 trunc from 0x828000 to 0x190af7 (0x697508 bytes) 000443[1] 1790920179.655443 punch from 0x0fe596 to 0x10a798 (0xc203 bytes) 000446[0] 1790920179.675643 mapread 0x030000 thru 0x037ddc (0x7ddd bytes) 000447[1] 1790920179.723280 write 0x387000 thru 0x394cf0 (0xdcf1 bytes) 000448[1] 1790920179.750977 punch from 0x33008a to 0x332254 (0x21cb bytes) 000449[0] 1790920179.786003 write_OD 0x644000 thru 0x64ffff (0xc000 bytes) 000451[1] 1790920179.902334 write 0x58d000 thru 0x591d3b (0x4d3c bytes) 000454[0] 1790920179.950986 mapread 0x4a3000 thru 0x4a75bc (0x45bd bytes) 000455[1] 1790920180.028010 punch from 0x470f12 to 0x47b3e2 (0xa4d1 bytes) 000460[1] 1790920180.128269 write 0x75c000 thru 0x7645c0 (0x85c1 bytes) 000461[0] 1790920180.152271 mapwrite 0x195000 thru 0x195dcf (0xdd0 bytes) 000462[1] 1790920180.315823 mapread 0x435000 thru 0x43e9fe (0x99ff bytes) 000464[0] 1790920180.369963 trunc from 0x7645c1 to 0x8fad4a (0x19678a bytes) 000465[0] 1790920180.450290 read_OD 0x00a000 thru 0x00afff (0x1000 bytes) 000467[1] 1790920180.484247 trunc from 0x8fad4b to 0x497872 (0x4634d8 bytes) 000468[0] 1790920180.551501 punch from 0x1b1095 to 0x1b49c1 (0x392d bytes) 000469[0] 1790920180.623353 trunc from 0x497873 to 0x20152c (0x296346 bytes) 000470[1] 1790920180.700867 punch from 0x0d49d0 to 0x0dcb6b (0x819c bytes) 000471[0] 1790920180.724882 read_OD 0x112000 thru 0x11afff (0x9000 bytes) 000472[1] 1790920180.830435 mapwrite 0x136000 thru 0x13cb1c (0x6b1d bytes) 000473[1] 1790920181.081219 mapread 0x0c3000 thru 0x0c5680 (0x2681 bytes) 000475[0] 1790920181.194664 trunc from 0x20152d to 0x8ccaea (0x6cb5be bytes) 000477[1] 1790920181.461103 read 0x56c000 thru 0x56cc5c (0xc5d bytes) 000478[1] 1790920181.530903 write 0x56b000 thru 0x56b6ab (0x6ac bytes) truncating to largest ever: 0x979578 000479[1] 1790920181.595684 trunc from 0x8ccaeb to 0x979577 (0xaca8d bytes) 000480[0] 1790920181.648013 mapwrite 0x72d000 thru 0x735824 (0x8825 bytes) 000481[0] 1790920182.290188 write_OD 0x763000 thru 0x766fff (0x4000 bytes) 000482[1] 1790920182.469637 write 0x6ba000 thru 0x6bad51 (0xd52 bytes) 000484[1] 1790920182.501237 mapwrite 0x172000 thru 0x1773cd (0x53ce bytes) 000485[1] 1790920182.784786 trunc from 0x979578 to 0x15a08c (0x81f4eb bytes) 000488[0] 1790920182.903667 mapread 0x12d000 thru 0x13ac37 (0xdc38 bytes) 000490[0] 1790920183.087737 write_OD 0x089000 thru 0x095fff (0xd000 bytes) 000491[1] 1790920183.169959 punch from 0x122c37 to 0x128fdf (0x63a9 bytes) 000492[0] 1790920183.194176 read_OD 0x03b000 thru 0x043fff (0x9000 bytes) 000493[1] 1790920183.219964 mapwrite 0x9dd000 thru 0x9e5ff0 (0x8ff1 bytes) 000494[1] 1790920183.412772 write 0x504000 thru 0x50b556 (0x7557 bytes) 000496[1] 1790920183.425173 mapread 0x0b8000 thru 0x0c2321 (0xa322 bytes) 000497[1] 1790920183.465183 read 0x5a0000 thru 0x5a3fdb (0x3fdc bytes) 000499[1] 1790920183.474962 mapread 0x58a000 thru 0x58bee1 (0x1ee2 bytes) 000500[0] 1790920183.527003 write_OD 0x2ec000 thru 0x2f2fff (0x7000 bytes) All operations completed A-OK! PASS 16a (66s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 01:49:49 (1790920189) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.51499 s, 16.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.355852 s, 23.6 MB/s Chance of close/open is 1 in 50 Seed set to 193 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1790920194.231808 mapwrite 0x93c239 thru 0x93e018 (0x1de0 bytes) 000002[1] 1790920194.448615 mapwrite 0x67f34c thru 0x67fe9a (0xb4f bytes) 000003[0] 1790920194.577469 punch from 0x8e36f2 to 0x8e40e5 (0x9f4 bytes) truncating to largest ever: 0xa70b4 000008[0] 1790920194.793982 trunc from 0x93e019 to 0x0a70b3 (0x896f65 bytes) 000009[0] 1790920194.885093 mapwrite 0x13e2b5 thru 0x13f3ca (0x1116 bytes) truncating to largest ever: 0x731453 000013[0] 1790920195.150580 trunc from 0x13f3cb to 0x731452 (0x5f2088 bytes) 000019[1] 1790920195.310495 read 0x24292c thru 0x24459d (0x1c72 bytes) 000024[1] 1790920195.375132 write 0x126eb5 thru 0x127884 (0x9d0 bytes) 000025[1] 1790920195.429880 punch from 0x3c0be4 to 0x3c0d54 (0x171 bytes) 000026[1] 1790920195.453672 read 0x16690b thru 0x168236 (0x192c bytes) 000027[0] 1790920195.475662 write 0x6a114f thru 0x6a2a61 (0x1913 bytes) 000028[0] 1790920195.504137 write 0x33357e thru 0x3347e1 (0x1264 bytes) truncating to largest ever: 0x8a3286 000029[0] 1790920195.534685 trunc from 0x731453 to 0x8a3285 (0x171e33 bytes) 000031[0] 1790920195.592761 mapwrite 0x42393c thru 0x4253e5 (0x1aaa bytes) 000034[1] 1790920195.805708 read 0x1a0643 thru 0x1a13a8 (0xd66 bytes) 000035[1] 1790920195.832133 trunc from 0x8a3286 to 0x2ef25b (0x5b402a bytes) 000036[0] 1790920195.892066 mapread 0x02605c thru 0x026705 (0x6aa bytes) 000038[1] 1790920196.022925 read 0x064dbd thru 0x065e83 (0x10c7 bytes) 000039[0] 1790920196.112284 mapread 0x14a1c9 thru 0x14a3d2 (0x20a bytes) 000040[1] 1790920196.148383 read 0x119899 thru 0x119b05 (0x26d bytes) 000041[1] 1790920196.162603 mapwrite 0x813001 thru 0x81495a (0x195a bytes) 000044[0] 1790920196.605787 punch from 0x3dc74a to 0x3dc995 (0x24c bytes) 000050[0] 1790920196.917136 write 0x8511ce thru 0x852c12 (0x1a45 bytes) 000052[0] 1790920196.994014 punch from 0x76e2ae to 0x76ef5d (0xcb0 bytes) 000056[1] 1790920197.163273 punch from 0x5059fb to 0x506a3c (0x1042 bytes) 000058[0] 1790920197.250763 mapread 0x6770d5 thru 0x678965 (0x1891 bytes) 000059[1] 1790920197.316325 punch from 0x3dddb9 to 0x3dee59 (0x10a1 bytes) 000061[1] 1790920197.436735 punch from 0x65722a to 0x6579be (0x795 bytes) 000062[0] 1790920197.459334 read 0x0c5d29 thru 0x0c6ae6 (0xdbe bytes) 000073[0] 1790920197.661510 mapread 0x79e591 thru 0x7a009d (0x1b0d bytes) 000074[0] 1790920197.752614 mapread 0x1a860c thru 0x1a89be (0x3b3 bytes) 000077[0] 1790920197.784962 trunc from 0x852c13 to 0x182d83 (0x6cfe8f bytes) 000079[1] 1790920197.938175 read 0x0c4dd9 thru 0x0c5242 (0x46a bytes) 000080[1] 1790920198.001368 read 0x09b7d7 thru 0x09b948 (0x172 bytes) 000081[1] 1790920198.017833 punch from 0x0b1cba to 0x0b34aa (0x17f1 bytes) 000083[0] 1790920198.096221 read 0x16477b thru 0x164f91 (0x817 bytes) 000085[0] 1790920198.172763 mapwrite 0x16027f thru 0x16076e (0x4f0 bytes) 000086[1] 1790920198.330244 mapread 0x13d80b thru 0x13e1d5 (0x9cb bytes) 000087[0] 1790920198.451860 trunc from 0x182d84 to 0x1a4503 (0x21780 bytes) 000089[0] 1790920198.519034 punch from 0x0d7b2f to 0x0d85e2 (0xab4 bytes) 000091[0] 1790920198.524660 mapread 0x0d1398 thru 0x0d29c3 (0x162c bytes) 000094[1] 1790920198.572301 punch from 0x0ef0db to 0x0ef2b5 (0x1db bytes) 000095[0] 1790920198.638356 punch from 0x03e769 to 0x03f604 (0xe9c bytes) 000096[1] 1790920198.701209 write 0x6e9094 thru 0x6e9997 (0x904 bytes) 000097[1] 1790920198.760624 write 0x5c8d83 thru 0x5c9222 (0x4a0 bytes) 000098[1] 1790920198.797210 write 0x5dfe99 thru 0x5e0021 (0x189 bytes) 000101[1] 1790920198.862580 trunc from 0x6e9998 to 0x36593b (0x38405c bytes) 000102[0] 1790920198.903141 mapread 0x2af8f9 thru 0x2b183c (0x1f44 bytes) 000103[0] 1790920199.021097 mapwrite 0x9da767 thru 0x9dc067 (0x1901 bytes) 000104[1] 1790920199.379611 mapwrite 0x97d50e thru 0x97d924 (0x417 bytes) 000106[1] 1790920199.631895 punch from 0x863803 to 0x863bd2 (0x3d0 bytes) 000110[0] 1790920199.739055 punch from 0x07d4e7 to 0x07ee85 (0x199f bytes) 000111[0] 1790920199.807925 mapread 0x7e77d5 thru 0x7e89ef (0x121b bytes) 000113[0] 1790920199.834327 punch from 0x311eee to 0x312cbd (0xdd0 bytes) 000114[0] 1790920199.856477 read 0x9c6843 thru 0x9c7f62 (0x1720 bytes) 000115[0] 1790920199.897667 mapwrite 0x087617 thru 0x088295 (0xc7f bytes) 000116[1] 1790920200.035299 trunc from 0x9dc068 to 0x4004b7 (0x5dbbb0 bytes) 000117[0] 1790920200.119640 mapwrite 0x8faa46 thru 0x8fc1e6 (0x17a1 bytes) 000118[1] 1790920200.308104 write 0x029179 thru 0x02aec2 (0x1d4a bytes) 000119[0] 1790920200.439514 trunc from 0x8fc1e7 to 0x5e3464 (0x318d82 bytes) 000120[0] 1790920200.542600 mapread 0x3b0997 thru 0x3b27b5 (0x1e1f bytes) 000121[0] 1790920200.620759 read 0x586e24 thru 0x587c2e (0xe0b bytes) 000126[1] 1790920200.643488 mapread 0x482da4 thru 0x4842a4 (0x1501 bytes) 000127[0] 1790920200.739993 read 0x4bd1e4 thru 0x4bf165 (0x1f82 bytes) 000128[0] 1790920200.763773 mapread 0x2a4bef thru 0x2a5f8c (0x139e bytes) 000129[1] 1790920200.775363 punch from 0x3be9cf to 0x3bec72 (0x2a4 bytes) 000130[1] 1790920200.810192 mapwrite 0x0374ff thru 0x038abf (0x15c1 bytes) 000133[0] 1790920200.919361 mapwrite 0x9400fe thru 0x941526 (0x1429 bytes) 000134[1] 1790920201.054872 punch from 0x44aa46 to 0x44c6da (0x1c95 bytes) 000138[0] 1790920201.150344 punch from 0x73567e to 0x736a72 (0x13f5 bytes) 000139[0] 1790920201.165840 trunc from 0x941527 to 0x40e73b (0x532deb bytes) 000140[0] 1790920201.217351 read 0x254644 thru 0x25571e (0x10db bytes) 000142[1] 1790920201.254878 read 0x28ea85 thru 0x28fe1d (0x1399 bytes) 000144[1] 1790920201.298002 write 0x27e37a thru 0x2802ed (0x1f74 bytes) 000145[0] 1790920201.365328 read 0x324e6e thru 0x326ae7 (0x1c7a bytes) 000146[0] 1790920201.394218 mapread 0x112d9e thru 0x113e14 (0x1077 bytes) 000147[0] 1790920201.436060 mapwrite 0x00744c thru 0x007ed5 (0xa8a bytes) 000148[1] 1790920201.538140 write 0x905a04 thru 0x906f1a (0x1517 bytes) 000150[0] 1790920201.612168 punch from 0x36a950 to 0x36afb1 (0x662 bytes) 000152[1] 1790920201.670916 mapread 0x29b91a thru 0x29c43e (0xb25 bytes) 000154[1] 1790920201.807807 punch from 0x7738d0 to 0x77446c (0xb9d bytes) 000156[1] 1790920201.890118 mapread 0x7af679 thru 0x7b0f32 (0x18ba bytes) 000158[1] 1790920201.908221 write 0x0eb952 thru 0x0ec6b3 (0xd62 bytes) 000160[0] 1790920202.018193 trunc from 0x906f1b to 0x16a54d (0x79c9cd bytes) 000162[0] 1790920202.139079 read 0x169625 thru 0x169f8c (0x968 bytes) 000163[0] 1790920202.186770 mapread 0x0f5fbe thru 0x0f75be (0x1601 bytes) 000164[0] 1790920202.217480 mapread 0x0eecaf thru 0x0f0ca3 (0x1ff5 bytes) 000165[1] 1790920202.243252 write 0x8a4bec thru 0x8a55c9 (0x9de bytes) 000167[1] 1790920202.304974 trunc from 0x8a55ca to 0x5eb0c8 (0x2ba501 bytes) 000172[0] 1790920202.398804 write 0x887fdb thru 0x889102 (0x1128 bytes) 000173[1] 1790920202.428082 trunc from 0x889103 to 0x2f8543 (0x590bbf bytes) 000174[1] 1790920202.479180 punch from 0x25d526 to 0x25f3c1 (0x1e9c bytes) 000175[1] 1790920202.511203 trunc from 0x2f8544 to 0x262f74 (0x955cf bytes) 000176[0] 1790920202.546807 trunc from 0x262f75 to 0x47f0f0 (0x21c17c bytes) 000177[0] 1790920202.710837 read 0x38f10c thru 0x390010 (0xf05 bytes) 000181[1] 1790920202.844949 trunc from 0x47f0f1 to 0x743a11 (0x2c4921 bytes) 000185[1] 1790920202.960819 trunc from 0x743a12 to 0x0d687b (0x66d196 bytes) 000186[0] 1790920203.004754 read 0x0d4fac thru 0x0d687b (0x18d0 bytes) 000187[0] 1790920203.051205 trunc from 0x0d687c to 0x0c7976 (0xef05 bytes) 000190[0] 1790920203.163532 trunc from 0x0c7977 to 0x402f1c (0x33b5a6 bytes) 000191[1] 1790920203.200820 trunc from 0x402f1d to 0x73ba46 (0x338b2a bytes) 000192[1] 1790920203.267229 mapread 0x5106e8 thru 0x511865 (0x117e bytes) 000197[1] 1790920203.382837 write 0x1a4186 thru 0x1a4546 (0x3c1 bytes) 000198[0] 1790920203.448014 read 0x4267b7 thru 0x426877 (0xc1 bytes) 000199[0] 1790920203.478106 write 0x97c50f thru 0x97d3e7 (0xed9 bytes) 000202[0] 1790920203.517962 read 0x2bf124 thru 0x2c00d8 (0xfb5 bytes) 000204[1] 1790920203.567800 write 0x390d75 thru 0x391794 (0xa20 bytes) 000205[0] 1790920203.640034 read 0x7f0d85 thru 0x7f2816 (0x1a92 bytes) 000207[0] 1790920203.680660 write 0x437326 thru 0x4374ed (0x1c8 bytes) 000208[0] 1790920203.722826 trunc from 0x97d3e8 to 0x6cbc69 (0x2b177e bytes) 000209[1] 1790920203.778185 mapwrite 0x280dfd thru 0x281f49 (0x114d bytes) 000210[0] 1790920203.892524 punch from 0x024cb4 to 0x02536d (0x6ba bytes) 000211[0] 1790920203.942257 mapwrite 0x42c29a thru 0x42de04 (0x1b6b bytes) 000212[1] 1790920204.005878 read 0x1bba73 thru 0x1bc261 (0x7ef bytes) 000214[0] 1790920204.054030 trunc from 0x6cbc6a to 0x7220c4 (0x5645b bytes) 000215[1] 1790920204.103038 trunc from 0x7220c5 to 0x0e09a1 (0x641723 bytes) 000218[0] 1790920204.219320 punch from 0x0ad761 to 0x0ae34a (0xbea bytes) 000219[1] 1790920204.285107 mapread 0x0d8e3c thru 0x0daa6c (0x1c31 bytes) 000220[1] 1790920204.327879 mapwrite 0x351aa4 thru 0x3531de (0x173b bytes) 000221[0] 1790920204.446390 mapwrite 0x9b1673 thru 0x9b27b9 (0x1147 bytes) 000222[0] 1790920204.692006 punch from 0x868ae1 to 0x869cf7 (0x1217 bytes) truncating to largest ever: 0x94cbf6 000226[0] 1790920204.836856 trunc from 0x9b27ba to 0x94cbf5 (0x65bc4 bytes) 000227[0] 1790920204.878547 trunc from 0x94cbf6 to 0x414e88 (0x537d6d bytes) 000229[0] 1790920204.924981 trunc from 0x414e89 to 0x356e6c (0xbe01c bytes) 000230[1] 1790920204.971356 read 0x106fc8 thru 0x1085ee (0x1627 bytes) 000231[1] 1790920205.092060 mapwrite 0x0dbf15 thru 0x0dc415 (0x501 bytes) 000232[1] 1790920205.349689 trunc from 0x356e6d to 0x6fa9ea (0x3a3b7e bytes) 000233[0] 1790920205.410844 read 0x488b1c thru 0x48a3d2 (0x18b7 bytes) 000235[0] 1790920205.456273 read 0x1889a1 thru 0x189b37 (0x1197 bytes) 000236[1] 1790920205.483552 read 0x4cc565 thru 0x4cd5a2 (0x103e bytes) 000239[1] 1790920205.498693 mapread 0x408107 thru 0x4086d3 (0x5cd bytes) 000240[1] 1790920205.519940 mapread 0x453be2 thru 0x4550b9 (0x14d8 bytes) 000242[0] 1790920205.564098 mapwrite 0x593b0a thru 0x595766 (0x1c5d bytes) 000243[0] 1790920205.717014 read 0x1fd029 thru 0x1fe5ea (0x15c2 bytes) 000245[1] 1790920205.728808 mapwrite 0x706eb1 thru 0x7074c6 (0x616 bytes) 000248[0] 1790920205.891386 read 0x51ea5f thru 0x52010d (0x16af bytes) 000249[1] 1790920205.927009 write 0x8512a8 thru 0x851d9a (0xaf3 bytes) 000250[1] 1790920205.957679 read 0x4ccd40 thru 0x4cce3d (0xfe bytes) 000252[0] 1790920206.005127 punch from 0x7144d0 to 0x715c1a (0x174b bytes) 000253[1] 1790920206.058815 mapread 0x0f2724 thru 0x0f2c41 (0x51e bytes) 000254[1] 1790920206.114023 read 0x6b56a1 thru 0x6b5927 (0x287 bytes) 000255[1] 1790920206.139070 mapread 0x49be58 thru 0x49be6f (0x18 bytes) 000256[1] 1790920206.145957 write 0x221cc4 thru 0x222ea3 (0x11e0 bytes) 000258[0] 1790920206.163089 write 0x61e801 thru 0x61e895 (0x95 bytes) 000259[1] 1790920206.203595 read 0x19ec87 thru 0x19f2c4 (0x63e bytes) 000262[0] 1790920206.325589 write 0x65d164 thru 0x65f095 (0x1f32 bytes) 000263[1] 1790920206.358206 mapread 0x6b2adc thru 0x6b303d (0x562 bytes) 000265[0] 1790920206.427288 mapread 0x19a149 thru 0x19a971 (0x829 bytes) 000266[1] 1790920206.496678 trunc from 0x851d9b to 0x8a77bd (0x55a23 bytes) 000267[1] 1790920206.570809 mapwrite 0x50107f thru 0x5011ef (0x171 bytes) 000268[0] 1790920206.695998 write 0x4128ae thru 0x413552 (0xca5 bytes) 000269[1] 1790920206.771592 mapwrite 0x657948 thru 0x658136 (0x7ef bytes) 000270[0] 1790920206.887303 write 0x072ed3 thru 0x073fdd (0x110b bytes) 000273[1] 1790920206.956881 mapread 0x5fe930 thru 0x6002b7 (0x1988 bytes) 000274[0] 1790920206.991660 punch from 0x5936f8 to 0x593700 (0x9 bytes) 000276[0] 1790920207.143384 trunc from 0x8a77be to 0x091a05 (0x815db8 bytes) 000278[0] 1790920207.287509 mapread 0x00b9e9 thru 0x00d90e (0x1f26 bytes) 000280[1] 1790920207.380139 punch from 0x0713a2 to 0x07294d (0x15ac bytes) 000282[1] 1790920207.474926 mapwrite 0x27c81a thru 0x27d0ea (0x8d1 bytes) 000284[0] 1790920207.589627 write 0x562ad1 thru 0x562f82 (0x4b2 bytes) 000286[0] 1790920207.635769 write 0x999006 thru 0x99adb7 (0x1db2 bytes) 000287[1] 1790920207.650361 punch from 0x461033 to 0x4623a7 (0x1375 bytes) 000288[1] 1790920207.684775 write 0x379559 thru 0x379e73 (0x91b bytes) 000291[0] 1790920207.784823 mapread 0x7c73ab thru 0x7c8aa1 (0x16f7 bytes) 000293[0] 1790920207.870184 punch from 0x7de649 to 0x7df9ea (0x13a2 bytes) 000294[0] 1790920207.932695 mapwrite 0x1b2fde thru 0x1b4b88 (0x1bab bytes) 000296[0] 1790920208.046864 read 0x3ec96c thru 0x3ed62d (0xcc2 bytes) 000297[0] 1790920208.082196 write 0x027a7f thru 0x027a84 (0x6 bytes) 000299[1] 1790920208.093724 punch from 0x79f869 to 0x7a0bc4 (0x135c bytes) 000302[1] 1790920208.181642 punch from 0x2b4c61 to 0x2b50c3 (0x463 bytes) 000304[0] 1790920208.213630 mapwrite 0x67533f thru 0x676881 (0x1543 bytes) 000306[0] 1790920208.382687 mapwrite 0x0fa79b thru 0x0fac4a (0x4b0 bytes) 000307[0] 1790920208.450612 mapwrite 0x11803a thru 0x118052 (0x19 bytes) 000308[1] 1790920208.529061 punch from 0x7d7559 to 0x7d80b2 (0xb5a bytes) 000309[1] 1790920208.566568 punch from 0x8ab491 to 0x8ad35a (0x1eca bytes) 000310[0] 1790920208.596192 punch from 0x7acb90 to 0x7ace2d (0x29e bytes) 000311[1] 1790920208.631283 write 0x791108 thru 0x792bb0 (0x1aa9 bytes) 000312[1] 1790920208.655687 punch from 0x7e2234 to 0x7e4004 (0x1dd1 bytes) 000314[1] 1790920208.691043 punch from 0x70e4c5 to 0x70e86f (0x3ab bytes) 000315[0] 1790920208.717080 write 0x447081 thru 0x448939 (0x18b9 bytes) 000316[0] 1790920208.755303 punch from 0x7971b4 to 0x798cae (0x1afb bytes) 000317[1] 1790920208.792676 write 0x171150 thru 0x171a26 (0x8d7 bytes) 000318[0] 1790920208.818813 mapwrite 0x50beb3 thru 0x50cc31 (0xd7f bytes) 000319[0] 1790920208.975647 mapread 0x5d5133 thru 0x5d637d (0x124b bytes) 000320[0] 1790920209.029663 mapread 0x463c06 thru 0x4651c1 (0x15bc bytes) 000321[0] 1790920209.117358 write 0x2415d7 thru 0x241c66 (0x690 bytes) 000322[1] 1790920209.190217 trunc from 0x99adb8 to 0x009c14 (0x9911a3 bytes) 000325[1] 1790920209.312205 read 0x00647b thru 0x007874 (0x13fa bytes) 000327[1] 1790920209.380920 trunc from 0x009c15 to 0x6f0bda (0x6e6fc6 bytes) 000328[0] 1790920209.433778 write 0x7d4e38 thru 0x7d5e87 (0x1050 bytes) 000332[0] 1790920209.524749 trunc from 0x7d5e88 to 0x4c49a3 (0x3114e4 bytes) 000335[1] 1790920209.621385 trunc from 0x4c49a4 to 0x600a6b (0x13c0c8 bytes) 000337[0] 1790920209.678066 write 0x3e9720 thru 0x3e9e53 (0x734 bytes) 000338[1] 1790920209.714608 punch from 0x18d3c2 to 0x18db6d (0x7ac bytes) 000339[1] 1790920209.741256 trunc from 0x600a6c to 0x038357 (0x5c8714 bytes) 000340[1] 1790920209.755787 mapwrite 0x450005 thru 0x450376 (0x372 bytes) 000342[1] 1790920209.908277 write 0x6cc768 thru 0x6cd8c7 (0x1160 bytes) 000343[1] 1790920209.913144 punch from 0x35ac3b to 0x35b570 (0x936 bytes) 000344[1] 1790920209.924587 mapread 0x1dc438 thru 0x1dc7d3 (0x39c bytes) 000345[1] 1790920209.925292 trunc from 0x6cd8c8 to 0x1dc4cb (0x4f13fc bytes) 000348[0] 1790920210.045289 mapwrite 0x073c15 thru 0x074f5c (0x1348 bytes) 000351[0] 1790920210.248765 mapwrite 0x1ede07 thru 0x1edfda (0x1d4 bytes) 000352[1] 1790920210.574081 read 0x0768c3 thru 0x077a24 (0x1162 bytes) 000353[0] 1790920210.727549 read 0x04a904 thru 0x04ab60 (0x25d bytes) 000356[1] 1790920210.880443 write 0x7a11ee thru 0x7a2956 (0x1769 bytes) 000357[0] 1790920210.926502 punch from 0x3bec53 to 0x3c0086 (0x1434 bytes) 000358[1] 1790920210.989334 trunc from 0x7a2957 to 0x0b972a (0x6e922c bytes) 000361[0] 1790920211.060505 punch from 0x06b3bf to 0x06b5b1 (0x1f3 bytes) 000364[1] 1790920211.132590 punch from 0x0b8dcc to 0x0b972a (0x95f bytes) 000365[0] 1790920211.181904 punch from 0x0a6146 to 0x0a6165 (0x20 bytes) 000367[1] 1790920211.227009 trunc from 0x0b972b to 0x64faf2 (0x5963c8 bytes) 000368[1] 1790920211.276925 mapread 0x606a85 thru 0x607fbc (0x1538 bytes) 000372[0] 1790920211.411681 mapwrite 0x764e26 thru 0x766575 (0x1750 bytes) 000373[0] 1790920211.660910 mapwrite 0x7da914 thru 0x7dc273 (0x1960 bytes) 000374[0] 1790920211.756943 mapwrite 0x10aeac thru 0x10af60 (0xb5 bytes) 000375[1] 1790920211.823426 mapwrite 0x30d9d1 thru 0x30eaca (0x10fa bytes) 000376[1] 1790920211.915951 mapread 0x68df56 thru 0x68e42a (0x4d5 bytes) 000377[1] 1790920211.999753 punch from 0x119d12 to 0x11b49c (0x178b bytes) 000378[0] 1790920212.027920 punch from 0x222a8a to 0x222b6d (0xe4 bytes) 000379[0] 1790920212.049315 read 0x488a70 thru 0x48949a (0xa2b bytes) 000380[0] 1790920212.061144 mapread 0x53e067 thru 0x53e925 (0x8bf bytes) 000381[0] 1790920212.072748 mapwrite 0x4fc711 thru 0x4fcc7d (0x56d bytes) 000382[1] 1790920212.203211 write 0x485eac thru 0x48605b (0x1b0 bytes) 000384[0] 1790920212.289668 mapread 0x05a5f7 thru 0x05af8d (0x997 bytes) 000388[1] 1790920212.383002 mapwrite 0x6eb1a3 thru 0x6eca71 (0x18cf bytes) 000390[0] 1790920212.492001 read 0x6ab634 thru 0x6ac6ee (0x10bb bytes) 000391[1] 1790920212.514304 mapwrite 0x76e968 thru 0x770124 (0x17bd bytes) 000392[0] 1790920212.641181 punch from 0x735836 to 0x736bf2 (0x13bd bytes) 000393[1] 1790920212.656977 mapwrite 0x32c22b thru 0x32d238 (0x100e bytes) 000394[1] 1790920212.803210 mapread 0x77aabe thru 0x77bb8c (0x10cf bytes) 000395[0] 1790920212.875783 mapwrite 0x72b25f thru 0x72b44d (0x1ef bytes) 000397[1] 1790920212.987859 mapwrite 0x016161 thru 0x0172e8 (0x1188 bytes) 000398[1] 1790920213.111204 mapread 0x691c04 thru 0x693aac (0x1ea9 bytes) 000399[1] 1790920213.170363 mapread 0x5cf087 thru 0x5d0053 (0xfcd bytes) 000400[0] 1790920213.198964 punch from 0x4a03f4 to 0x4a2084 (0x1c91 bytes) 000402[1] 1790920213.251805 mapwrite 0x2537e6 thru 0x2556f7 (0x1f12 bytes) 000405[0] 1790920213.358889 read 0x0c7a90 thru 0x0c7f36 (0x4a7 bytes) 000406[0] 1790920213.375123 trunc from 0x7dc274 to 0x4e8622 (0x2f3c51 bytes) 000410[0] 1790920213.462210 write 0x979ac4 thru 0x97acf5 (0x1232 bytes) 000411[1] 1790920213.493688 read 0x7f6eea thru 0x7f75f6 (0x70d bytes) 000413[1] 1790920213.554677 read 0x36f1ed thru 0x370a99 (0x18ad bytes) 000414[0] 1790920213.575635 read 0x09fb8c thru 0x0a1ac0 (0x1f35 bytes) 000415[0] 1790920213.588544 write 0x022fb0 thru 0x024085 (0x10d6 bytes) 000416[0] 1790920213.626352 trunc from 0x97acf6 to 0x40f24e (0x56baa7 bytes) 000418[0] 1790920213.716899 punch from 0x0e4a67 to 0x0e5421 (0x9bb bytes) 000419[0] 1790920213.742710 punch from 0x3cfe2a to 0x3cfea7 (0x7e bytes) 000420[1] 1790920213.770613 write 0x59a4ce thru 0x59add0 (0x903 bytes) 000425[0] 1790920213.880548 mapread 0x19c8de thru 0x19cec1 (0x5e4 bytes) 000430[0] 1790920214.043938 mapread 0x506393 thru 0x5064ed (0x15b bytes) 000432[1] 1790920214.141302 read 0x32cd7f thru 0x32e907 (0x1b89 bytes) 000433[0] 1790920214.190930 read 0x1676e9 thru 0x1694b7 (0x1dcf bytes) 000434[0] 1790920214.200310 read 0x422e1b thru 0x4237ce (0x9b4 bytes) 000435[1] 1790920214.216316 write 0x52f2c6 thru 0x53018b (0xec6 bytes) 000438[0] 1790920214.252354 trunc from 0x59add1 to 0x51d331 (0x7da9f bytes) 000439[0] 1790920214.340960 trunc from 0x51d332 to 0x3fb608 (0x121d29 bytes) 000440[1] 1790920214.357753 mapwrite 0x48ab00 thru 0x48c2a0 (0x17a1 bytes) 000441[1] 1790920214.613456 trunc from 0x48c2a1 to 0x6f2c80 (0x2669e0 bytes) 000444[0] 1790920214.739923 write 0x70a6ed thru 0x70b90e (0x1222 bytes) 000446[1] 1790920214.780723 punch from 0x218f1d to 0x21ac27 (0x1d0b bytes) 000447[1] 1790920214.803167 read 0x66e209 thru 0x66e3bf (0x1b7 bytes) 000448[0] 1790920214.827951 punch from 0x55f26b to 0x560d54 (0x1aea bytes) 000449[0] 1790920214.855912 read 0x3df77a thru 0x3e108b (0x1912 bytes) 000453[1] 1790920214.911473 mapwrite 0x081269 thru 0x082ead (0x1c45 bytes) 000454[1] 1790920215.008443 read 0x092461 thru 0x092663 (0x203 bytes) 000455[0] 1790920215.025247 read 0x4299e5 thru 0x42b4eb (0x1b07 bytes) 000456[1] 1790920215.044300 punch from 0x47b939 to 0x47be78 (0x540 bytes) 000458[0] 1790920215.083951 mapread 0x3d3f5d thru 0x3d402c (0xd0 bytes) 000460[0] 1790920215.120376 punch from 0x1fb7a9 to 0x1fbe56 (0x6ae bytes) 000463[1] 1790920215.183959 punch from 0x249c93 to 0x24b113 (0x1481 bytes) 000464[0] 1790920215.217899 mapread 0x20059e thru 0x200aec (0x54f bytes) 000465[1] 1790920215.247219 read 0x6f5793 thru 0x6f6e5a (0x16c8 bytes) 000468[0] 1790920215.254189 read 0x0d9a08 thru 0x0db787 (0x1d80 bytes) 000470[0] 1790920215.359985 mapread 0x3ee10a thru 0x3eeae9 (0x9e0 bytes) 000471[0] 1790920215.369198 read 0x622332 thru 0x623b0f (0x17de bytes) 000473[0] 1790920215.389591 punch from 0x37e80b to 0x37faff (0x12f5 bytes) 000474[1] 1790920215.415609 write 0x5dd9d1 thru 0x5ddaa0 (0xd0 bytes) 000475[0] 1790920215.449394 write 0x315121 thru 0x31538a (0x26a bytes) 000477[0] 1790920215.479120 mapread 0x014b66 thru 0x015cee (0x1189 bytes) 000478[1] 1790920215.507537 mapread 0x3b8c0c thru 0x3b9321 (0x716 bytes) 000480[1] 1790920215.575315 mapread 0x3bec10 thru 0x3becee (0xdf bytes) 000481[1] 1790920215.585988 trunc from 0x70b90f to 0x630d49 (0xdabc5 bytes) 000482[0] 1790920215.620177 mapwrite 0x074b98 thru 0x07643c (0x18a5 bytes) 000483[1] 1790920215.774191 mapwrite 0x8fe49c thru 0x8ffe94 (0x19f9 bytes) 000485[1] 1790920215.929624 punch from 0x636f09 to 0x637192 (0x28a bytes) 000489[1] 1790920215.994996 trunc from 0x8ffe95 to 0x14be21 (0x7b4073 bytes) 000490[1] 1790920216.019014 read 0x0bf3fc thru 0x0c07b8 (0x13bd bytes) 000491[0] 1790920216.079978 mapwrite 0x7f8177 thru 0x7f8aaa (0x934 bytes) 000494[0] 1790920216.276470 punch from 0x1d9344 to 0x1d9cde (0x99b bytes) 000496[1] 1790920216.312303 punch from 0x733600 to 0x7351ec (0x1bed bytes) 000497[0] 1790920216.352803 write 0x1f4bb2 thru 0x1f5c4d (0x109c bytes) 000498[0] 1790920216.379775 trunc from 0x7f8aab to 0x031896 (0x7c7214 bytes) 000500[1] 1790920216.527742 punch from 0x008c86 to 0x009f86 (0x1301 bytes) All operations completed A-OK! PASS 16b (33s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 01:50:22 (1790920222) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 01:50:27 (1790920227) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.534281 s, 15.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.477809 s, 17.6 MB/s ldlm.namespaces.MGC192.168.203.118@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8d958326b000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8d959031d800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8d958326b000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8d959031d800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8d958326b000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8d958326b000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7805 s, 8.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.2396 s, 32.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.46557 s, 42.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.8981 s, 17.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.80468 s, 13.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.41883 s, 30.7 MB/s PASS 16d (47s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 01:51:14 (1790920274) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.263679 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13716 s, 7.6 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 01:51:21 (1790920281) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 01:51:49 (1790920309) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 01:52:16 (1790920336) striped dir -i0 -c2 -H crush /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 01:52:25 (1790920345) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (9s) == sanityn test 16j: race dio with buffered i/o ========== 01:52:34 (1790920354) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.23407 s, 11.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.62908 s, 16.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.82858 s, 9.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.55362 s, 16.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.25397 s, 11.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.25555 s, 11.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.98871 s, 8.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.73156 s, 9.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.78009 s, 9.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.28446 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.60222 s, 7.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.26077 s, 11.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.41926 s, 10.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.92829 s, 9.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.14121 s, 8.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.39731 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.75263 s, 9.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.06852 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.45656 s, 7.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (40s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 01:53:15 (1790920395) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Oct 2 01:54:03 EDT 2026 + fsx 2 done: rc=0 Fri Oct 2 01:54:03 EDT 2026 + fsx 3 done: rc=0 Fri Oct 2 01:54:03 EDT 2026 + fsx 4 done: rc=0 Fri Oct 2 01:54:03 EDT 2026 + fsx 5 done: rc=0 Fri Oct 2 01:54:03 EDT 2026 PASS 16k (59s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 01:54:15 (1790920455) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 01:54:27 (1790920467) mmap test1: basic mmap operation (PASS, 0.17499s) mmap test2: MAP_PRIVATE not write back (PASS, 0.087681s) mmap test3: concurrent mmap ops on two nodes (PASS, 7.3229s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6382s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.10898s) mmap test6: check mmap write/read content on two nodes (PASS, 0.17074s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.1265s) mmap test11: multi-thread shared mmap (PASS, 0.99867s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0897s) PASS 18 (25s) == sanityn test 19: test concurrent uncached read races ========================================================================= 01:54:52 (1790920492) oleg318-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 01:54:57 (1790920497) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d20.sanityn PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 01:55:05 (1790920505) striped dir -i1 -c2 -H crush /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 01:55:11 (1790920511) now is 1790920513 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 new atime is 1790920574 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 01:56:21 (1790920581) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 519.6K 404 519.2K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 518.8K 336 518.5K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 115.4K 468 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 465 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 519.6K 404 519.2K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 518.8K 336 518.5K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 115.4K 468 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 465 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.4M 2.1G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 532116 404 531712 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 531300 336 530964 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 118132 468 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 465 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 235236 740 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 519.6K 404 519.2K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 518.8K 336 518.5K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 115.4K 468 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 465 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 532116 404 531712 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 531300 336 530964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 118132 468 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117297 465 116832 1% /mnt/lustre[OST:1] filesystem_summary: 235236 740 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 532116 404 531712 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 531300 336 530964 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 118132 468 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 465 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 235236 740 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 01:56:29 (1790920589) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 01:56:35 (1790920595) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 01:56:44 (1790920604) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (7s) == sanityn test 26a: allow mtime to get older ============ 01:56:50 (1790920610) utime: good mknod times 1790920611 <= 1790920612 <= 1790920612 for /mnt/lustre/f26a utime: good mknod times 1790920611 <= 1790920612 <= 1790920612 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 01:56:57 (1790920617) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 01:57:06 (1790920626) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279600 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-10-02 01:57:07.000000000 -0400 Birth: 2026-10-02 01:57:08.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279600 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-10-02 01:57:07.000000000 -0400 Birth: 2026-10-02 01:57:08.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 01:57:15 (1790920635) striped dir -i0 -c2 -H all_char /mnt/lustre/d30.sanityn 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 01:57:24 (1790920644) striped dir -i1 -c2 -H all_char /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 01:57:32 (1790920652) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg318-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete striped dir -i1 -c2 -H crush /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.118@tcp:/lustre 7542784 40960 7495680 1% /mnt/lustre2 PASS 31b (23s) == sanityn test 31r: open-rename(replace) race =========== 01:57:55 (1790920675) -rw-r--r-- 1 root root 0 Oct 2 01:57 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 01:58:05 (1790920685) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1211: 45212 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 01:58:13 (1790920693) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 01:58:22 (1790920702) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 01:58:27 (1790920707) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.83782 s, 28.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.31296 s, 39.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.20351 s, 23.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.55535 s, 20.5 MB/s contention_seconds 4, contention_events 17 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.09844 s, 25.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.31236 s, 22.7 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.40872 s, 37.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.44482 s, 36.3 MB/s PASS 32c (31s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 01:59:00 (1790920740) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 01:59:04 (1790920744) Stopping /mnt/lustre-mds1 (opts:) on oleg318-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (57s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 02:00:02 (1790920802) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 1 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (57s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 02:00:58 (1790920858) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 1.63 seconds: 61.19 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 1.35 seconds: 74.04 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1790920866 ; total 0 ; last 0) total: 100 unlinks in 7 seconds: 14.285714 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1790920873 ; total 0 ; last 0) total: 100 unlinks in 3 seconds: 33.333332 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (24s) == sanityn test 34: no lock timeout under IO ============= 02:01:22 (1790920882) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d958326b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d958326b000.ost_server_uuid in IDLE state after 0 sec oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d958326b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d958326b000.ost_server_uuid in FULL state after 0 sec oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d958326b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d958326b000.ost_server_uuid in IDLE state after 0 sec oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d958326b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d958326b000.ost_server_uuid in FULL state after 0 sec oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d958326b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d958326b000.ost_server_uuid in IDLE state after 0 sec oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d958326b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d958326b000.ost_server_uuid in FULL state after 0 sec PASS 34 (128s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 02:03:30 (1790921010) striped dir -i1 -c2 -H crush /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 57258 57337 for 60 sec... - open/close 719 (time 1790921026.46 total 10.01 last 71.81) - open/close 1352 (time 1790921036.47 total 20.02 last 63.25) - open/close 2094 (time 1790921046.47 total 30.02 last 74.20) - open/close 2927 (time 1790921056.47 total 40.02 last 83.28) - open/close 3729 (time 1790921066.47 total 50.02 last 80.19) total: 4000 open/close in 53.47 seconds: 74.81 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.118@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.203.118@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 02:04:41 (1790921081) striped dir -i0 -c2 -H crush /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg318-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.72207 s, 30.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.38458 s, 37.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4328 s, 36.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2398 s, 42.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.54135 s, 34.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7492608) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05451 s, 49.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.16293 s, 45.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.995132 s, 52.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7492608) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.21888 s, 43.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34874 s, 38.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(9) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.25775 s, 41.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7441408) after(7492608) PASS 36 (296s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 02:09:37 (1790921377) striped dir -i1 -c2 -H all_char /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 - create 1464 (time 1790921392.84 total 10.01 last 146.32) - create 2914 (time 1790921402.85 total 20.01 last 144.92) - create 4490 (time 1790921412.85 total 30.01 last 157.59) - create 5872 (time 1790921422.86 total 40.02 last 138.06) - create 7144 (time 1790921432.87 total 50.03 last 127.09) - create 8566 (time 1790921442.87 total 60.03 last 142.16) - create 9852 (time 1790921452.87 total 70.03 last 128.57) total: 10000 create in 71.19 seconds: 140.47 ops/second PASS 37 (83s) == sanityn test 39a: file mtime does not change after rename ========================================================== 02:11:00 (1790921460) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 02:11:07 (1790921467) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 02:11:15 (1790921475) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 02:11:23 (1790921483) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 02:11:31 (1790921491) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 02:11:49 (1790921509) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 02:12:05 (1790921525) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 02:12:20 (1790921540) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (15s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 02:12:35 (1790921555) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 02:12:47 (1790921567) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 02:13:00 (1790921580) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (15s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 02:13:15 (1790921595) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 02:13:28 (1790921608) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 02:13:41 (1790921621) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 02:13:53 (1790921633) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 02:14:05 (1790921645) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2433: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 02:14:18 (1790921658) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1049s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 02:31:48 (1790922708) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (15s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 02:32:03 (1790922723) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 02:32:18 (1790922738) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 02:32:31 (1790922751) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 02:32:45 (1790922765) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 02:32:58 (1790922778) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42f (11s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 02:33:10 (1790922790) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 02:33:22 (1790922802) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42h (12s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 02:33:34 (1790922814) PASS 43a (184s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 02:36:38 (1790922998) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43b (12s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 02:36:50 (1790923010) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43c (12s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 02:37:02 (1790923022) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 02:37:16 (1790923036) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 02:37:30 (1790923050) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (16s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 02:37:46 (1790923066) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 02:38:00 (1790923080) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 02:38:13 (1790923093) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43i (12s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 02:38:25 (1790923105) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (134s) == sanityn test 43k: unlink vs create ==================== 02:40:39 (1790923239) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1170s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 03:00:09 (1790924409) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Oct 2 03:00:16 EDT 2026 PASS 44a (11s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 03:00:21 (1790924421) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 44b (12s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 03:00:33 (1790924433) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (12s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 03:00:46 (1790924446) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44d (12s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 03:00:58 (1790924458) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44e (13s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 03:01:11 (1790924471) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44f (11s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 03:01:23 (1790924483) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44g (11s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 03:01:34 (1790924494) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44h (17s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 03:01:51 (1790924511) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (14s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 03:02:05 (1790924525) PASS 45a (205s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 03:05:30 (1790924730) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45b (11s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 03:05:41 (1790924741) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 03:05:54 (1790924754) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (12s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 03:06:06 (1790924766) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45e (13s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 03:06:19 (1790924779) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (13s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 03:06:33 (1790924793) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (13s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 03:06:45 (1790924805) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (12s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 03:06:57 (1790924817) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45i (11s) == sanityn test 45j: read vs rename ====================== 03:07:08 (1790924828) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1201s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 03:27:09 (1790926029) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 03:27:22 (1790926042) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 46b (11s) == sanityn test 46c: pdirops: link vs link ======================================================================== 03:27:33 (1790926053) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 03:27:45 (1790926065) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46d (14s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 03:28:00 (1790926080) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46e (16s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 03:28:15 (1790926095) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 03:28:28 (1790926108) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46g (12s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 03:28:40 (1790926120) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46h (13s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 03:28:53 (1790926133) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (12s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 03:29:05 (1790926145) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (13s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 03:29:18 (1790926158) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 47b (13s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 03:29:32 (1790926172) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (12s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 03:29:44 (1790926184) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47d (12s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 03:29:56 (1790926196) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (16s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 03:30:12 (1790926212) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47f (14s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 03:30:27 (1790926227) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (18s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 03:30:44 (1790926244) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.106644 s, 96.0 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 03:30:56 (1790926256) short read: 0 ->+ 158 -> 0 0 0+1 records in 0+1 records out 158 bytes copied, 0.0637198 s, 2.5 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3696: kill: (236767) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 03:31:06 (1790926266) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00664379 s, 154 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057336 s, 18.3 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 03:31:27 (1790926287) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0296846 s, 34.5 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 03:31:40 (1790926300) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788668 s, 13.3 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 03:31:49 (1790926309) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1280 0x500 0x2c0000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 1280 0x500 0x2c0000401 read lease(1) has applied. PASS 51e (12s) == sanityn test 54: rename locking ======================= 03:32:01 (1790926321) llite.lustre-ffff8d958326b000.xattr_cache=1 llite.lustre-ffff8d959031d800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762675 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 03:32:04.000000000 -0400 Modify: 2026-10-02 03:32:04.000000000 -0400 Change: 2026-10-02 03:32:04.000000000 -0400 Birth: 2026-10-02 03:32:05.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281271 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 03:32:11.000000000 -0400 Modify: 2026-10-02 03:32:13.000000000 -0400 Change: 2026-10-02 03:32:13.000000000 -0400 Birth: 2026-10-02 03:32:13.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762678 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 03:32:18.000000000 -0400 Modify: 2026-10-02 03:32:19.000000000 -0400 Change: 2026-10-02 03:32:19.000000000 -0400 Birth: 2026-10-02 03:32:20.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281272 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 03:32:18.000000000 -0400 Modify: 2026-10-02 03:32:20.000000000 -0400 Change: 2026-10-02 03:32:20.000000000 -0400 Birth: 2026-10-02 03:32:20.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762679 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 03:32:25.000000000 -0400 Modify: 2026-10-02 03:32:27.000000000 -0400 Change: 2026-10-02 03:32:27.000000000 -0400 Birth: 2026-10-02 03:32:27.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 14 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281274 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 03:32:25.000000000 -0400 Modify: 2026-10-02 03:32:27.000000000 -0400 Change: 2026-10-02 03:32:27.000000000 -0400 Birth: 2026-10-02 03:32:27.000000000 -0400 PASS 54 (36s) == sanityn test 55a: rename vs unlink target dir ========= 03:32:37 (1790926357) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 03:32:51 (1790926371) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 03:33:04 (1790926384) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (18s) == sanityn test 55d: rename file vs link ================= 03:33:23 (1790926403) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 03:33:37 (1790926417) fail_loc=0x155 We survived after AB/BA race test fail_loc=0x0 PASS 55e (19s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 03:33:57 (1790926437) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 03:34:17 (1790926457) mdt.lustre-MDT0000.enable_rename_trylock=1 mdt.lustre-MDT0001.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (23s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 03:34:40 (1790926480) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1790926482 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.586914 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (112s) SKIP: sanityn test_56b skipping ALWAYS excluded test 56b == sanityn test 60: Verify data_version behaviour ======== 03:36:34 (1790926594) striped dir -i0 -c2 -H crush /mnt/lustre/d60.sanityn PASS 60 (7s) == sanityn test 70a: cd directory && rm directory ======== 03:36:42 (1790926602) PASS 70a (6s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 03:36:48 (1790926608) PASS 70b (7s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 03:36:55 (1790926615) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0201376 s, 3.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 03:36:59 (1790926619) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0175209 s, 3.7 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 03:37:02 (1790926622) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 03:37:05 (1790926625) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0108413 s, 6.0 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 03:37:08 (1790926628) llite.lustre-ffff8d958326b000.xattr_cache=1 llite.lustre-ffff8d959031d800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (6s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 03:37:14 (1790926634) llite.lustre-ffff8d958326b000.xattr_cache=1 llite.lustre-ffff8d959031d800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 03:37:21 (1790926641) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0177637 s, 57.6 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0213443 s, 48.0 kB/s 249617: get lock2 249618: get lock1 249617: done 249617: get lock1 249618: done 249618: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 249618: cannot get lock2: Resource deadlock avoided 249618: exit rc=0 249617: done 249617: exit rc=0 PASS 74 (10s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 03:37:31 (1790926651) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0627072 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441156 s, 23.8 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4379: 250207 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 03:37:43 (1790926663) striped dir -i0 -c2 -H crush /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (180s) == sanityn test 77a: check FIFO NRS policy =============== 03:40:43 (1790926843) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246174 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285038 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110019 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118581 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098883 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147017 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196066 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180672 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185451 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162909 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203786 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233162 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200348 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220922 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247673 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262305 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311461 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471104 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245485 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391185 s, 26.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253771 s, 41.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274448 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450048 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317281 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422763 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359484 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068627 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425586 s, 24.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885649 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859446 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488503 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582455 s, 18.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670857 s, 15.6 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (8s) == sanityn test 77b: check CRR-N NRS policy ============== 03:40:51 (1790926851) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.442425 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823326 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811618 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11471 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217025 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265014 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27376 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247849 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223784 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287409 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259423 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320171 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273204 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307684 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329492 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273714 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397121 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732895 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361225 s, 29.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00480899 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104312 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135893 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156122 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108797 s, 96.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00950557 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00523743 s, 200 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208634 s, 50.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109456 s, 95.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161228 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141341 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665778 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159988 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825945 s, 12.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.764041 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712601 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959841 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638666 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184924 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185162 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246361 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320786 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196539 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209332 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239554 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235513 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329997 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313578 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213698 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246643 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286957 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241304 s, 435 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241101 s, 435 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00695805 s, 151 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00921707 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102947 s, 102 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815737 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103048 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543365 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172849 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00597913 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727368 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327364 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211124 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516941 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215294 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00934217 s, 112 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (15s) == sanityn test 77c: check ORR NRS policy ================ 03:41:06 (1790926866) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.409329 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564468 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684218 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989916 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101404 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121499 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180403 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210102 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201093 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28812 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313201 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237406 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241369 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27354 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302596 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296551 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334375 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978735 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725464 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111936 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544152 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117053 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05569 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591291 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654532 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748563 s, 14.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735332 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076692 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793315 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849418 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910426 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094872 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104772 s, 10.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.466109 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136237 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158065 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156978 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276594 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260589 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232223 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268774 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245323 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343016 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3681 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346018 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318206 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366931 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431058 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390662 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713157 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672774 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712938 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721649 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723125 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589991 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684129 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769882 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775849 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459926 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565145 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790251 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585399 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556154 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519042 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167149 s, 62.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.537605 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352239 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119633 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103858 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147197 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104027 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106999 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172256 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109138 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199913 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196531 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227126 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222484 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317688 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329295 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211854 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400416 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419162 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341249 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486767 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297511 s, 35.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919777 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810722 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760743 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837558 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305896 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106207 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059347 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496375 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510277 s, 20.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898931 s, 11.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063441 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092397 s, 11.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (18s) == sanityn test 77d: check TRR nrs policy ================ 03:41:25 (1790926885) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.585811 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742598 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608935 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178579 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167986 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133495 s, 7.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196937 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177164 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238315 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200064 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161232 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225002 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163338 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228656 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276658 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252932 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300621 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447547 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561841 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138279 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769127 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606104 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980496 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823556 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642076 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874283 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144961 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14562 s, 7.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111595 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148147 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114775 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150578 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114851 s, 9.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.760596 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457008 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351478 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640704 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113824 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152746 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256422 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282719 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274672 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260208 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237248 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290193 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271203 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281154 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252464 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314748 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335838 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835056 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556441 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874034 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164258 s, 63.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106713 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171838 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129849 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141644 s, 7.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00498821 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145633 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148713 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150876 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106416 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241903 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257712 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18257 s, 5.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.56444 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899137 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114618 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765495 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148302 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213893 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172952 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232968 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214999 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281334 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254258 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298167 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333451 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271899 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349166 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254867 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372893 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694195 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00574752 s, 182 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263256 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017199 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726715 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386762 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865872 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405646 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954569 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532048 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955855 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510972 s, 20.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566923 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979202 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648492 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632136 s, 16.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=7 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 03:41:43 (1790926903) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.18@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.600789 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631464 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166706 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174645 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157938 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928487 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235005 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228789 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245382 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148292 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307165 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195338 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276693 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303643 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252889 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23165 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331889 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815737 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104183 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943169 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163745 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168532 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134003 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175729 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138219 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140754 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129807 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14196 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119518 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147587 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159868 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155477 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169354 s, 6.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=3 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.519779 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761041 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103013 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101255 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12444 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135692 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248163 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294268 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324926 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23919 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340291 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230601 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358321 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293282 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256351 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372951 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400448 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07021 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462684 s, 22.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360212 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692051 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00802516 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01194 s, 87.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189116 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00656794 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216829 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00512142 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729259 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00958381 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608683 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506527 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292889 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00943425 s, 111 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.413784 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922201 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888715 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124461 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815934 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201696 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209781 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21662 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894487 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2084 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218487 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253047 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19518 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262285 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106864 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236646 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241218 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931032 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061705 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643797 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684318 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389576 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669004 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292466 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370189 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606712 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613052 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883741 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685926 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657682 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573838 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825201 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604247 s, 17.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.464827 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698334 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943756 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161718 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194793 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194029 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846909 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172042 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202288 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950077 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162258 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22347 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258192 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25862 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192921 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31086 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281609 s, 3.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035582 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669162 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747422 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058574 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610866 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568876 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105757 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757574 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887683 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764914 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772371 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839336 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835397 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876486 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931067 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969168 s, 10.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (24s) == sanityn test 77f: check TBF JobID nrs policy ========== 03:42:07 (1790926927) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.529011 s, 31.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798134 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988236 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114945 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157223 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212848 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312316 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340747 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363523 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299858 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309292 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367236 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27112 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358827 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319858 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34799 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327428 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00896619 s, 117 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162157 s, 64.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174733 s, 60.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202055 s, 51.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157274 s, 66.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137885 s, 7.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203845 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173603 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00721244 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699545 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572246 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256735 s, 40.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179173 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390819 s, 26.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072566 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107484 s, 9.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.523218 s, 32.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049025 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787109 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103943 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124183 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150944 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14428 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274268 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13824 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189853 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179321 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231352 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253935 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261078 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227504 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155578 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24644 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576122 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692831 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783902 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665523 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882233 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142699 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147593 s, 7.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132182 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124896 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115112 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103202 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109782 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110858 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966768 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130921 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12276 s, 8.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.513379 s, 32.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053334 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100472 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901979 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12393 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11827 s, 8.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202276 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00533032 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27328 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06174 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887523 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394903 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391269 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404154 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405984 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433322 s, 2.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374652 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351225 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436127 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364445 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00889819 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00934391 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00695134 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105499 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134272 s, 78.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222467 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201681 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938433 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104813 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00436913 s, 240 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.008257 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19263 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067552 s, 15.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.484513 s, 34.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635662 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803474 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128478 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792629 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157161 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109259 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125415 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908158 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256368 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321825 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337323 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294348 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266107 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26743 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268911 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42657 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896979 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115177 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116549 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101204 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937277 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119641 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857655 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129952 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103482 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124082 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157145 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102869 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127764 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147334 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148048 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108429 s, 9.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (25s) == sanityn test 77g: Change TBF type directly ============ 03:42:32 (1790926952) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (12s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 03:42:44 (1790926964) oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 PASS 77h (15s) == sanityn test 77i: Change rank of TBF rule ============= 03:42:59 (1790926979) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (27s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 03:43:26 (1790927006) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.12639 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981744 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934854 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178584 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261155 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37792 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290306 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252058 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142938 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41041 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429201 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262539 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341715 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330534 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315337 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281035 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380308 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949463 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647308 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538749 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111297 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087518 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785406 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102979 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616559 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850448 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124064 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696467 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966275 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102146 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894885 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105646 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102007 s, 10.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.46725 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6368 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (68s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 03:44:34 (1790927074) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30692 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102842 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145624 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162995 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175662 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199488 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212611 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248069 s, 4.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228823 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229924 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28512 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279883 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258317 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323208 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343428 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300514 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381289 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860966 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819982 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062567 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857025 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941539 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088823 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793344 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810768 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081883 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100642 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923393 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825406 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930056 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104777 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109153 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112087 s, 9.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9956 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0227 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33554 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829439 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152977 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159814 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136918 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189511 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111246 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249885 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119192 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250587 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115162 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165533 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26165 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280075 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201725 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244857 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248229 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087335 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961299 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088778 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931641 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107958 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857344 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957317 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822223 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103375 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169559 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930284 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097496 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089418 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17831 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184439 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171152 s, 6.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.552 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7656 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (159s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 03:47:13 (1790927233) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32799 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887772 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121499 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793574 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168661 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09175 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177845 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174865 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180016 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194391 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30086 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200916 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277439 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29732 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278839 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305586 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283941 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727368 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567511 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871355 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786701 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708566 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102022 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540886 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375468 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742413 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116178 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111615 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00726846 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00951119 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00657822 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261182 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00801689 s, 131 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0285 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8231 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3473 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22001 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154357 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254688 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183035 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279866 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410585 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164257 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495991 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288065 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.589309 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.581269 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.550319 s, 1.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.617833 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.524196 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.561051 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521364 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868602 s, 12.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871221 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697868 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551337 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889965 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110092 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082914 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892502 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131545 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110615 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1434 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112006 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118769 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102381 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118189 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136212 s, 7.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0584 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9567 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (154s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 03:49:47 (1790927387) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.595569 s, 28.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728404 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088045 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566324 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277205 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294012 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295462 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286294 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337637 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323594 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322175 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347999 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356695 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338348 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349092 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337713 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346096 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00300003 s, 350 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00250286 s, 419 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111158 s, 94.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105446 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013621 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014027 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00601027 s, 174 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00764767 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864441 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375002 s, 28.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139528 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121321 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296554 s, 35.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575049 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012464 s, 84.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00514944 s, 204 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2998 s, 10.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97858 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.18@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.18@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.685865 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561092 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812733 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232583 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233075 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223383 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238282 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256092 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238643 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250907 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292549 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299516 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293558 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309685 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297574 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311113 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350343 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998312 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166934 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106594 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129549 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11038 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119541 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162362 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145546 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173392 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132684 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137544 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1425 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146855 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14131 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148997 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198811 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=3 removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.06692 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94993 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.18@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.479294 s, 35.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515202 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818389 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679319 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220694 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230135 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196329 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199243 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203276 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221397 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223665 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196569 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224251 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250219 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232815 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219017 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275827 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108718 s, 96.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239232 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356349 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560165 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00285962 s, 367 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173885 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00311529 s, 337 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404315 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00412948 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569253 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029835 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285707 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275489 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00782702 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186895 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146193 s, 71.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.43342 s, 11.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.42131 s, 19.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.747434 s, 22.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485756 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718177 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799065 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372776 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118064 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155379 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19216 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126566 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168039 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213635 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189285 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253275 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255068 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257881 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291438 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295057 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362049 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222973 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655068 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492555 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492203 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329724 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109191 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534398 s, 19.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053417 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690631 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721304 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069503 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742329 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898117 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088565 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117421 s, 8.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1102 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90385 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.10896 s, 14.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.875 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.0292 s, 16.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705691 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808677 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170381 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206905 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191433 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256454 s, 4.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25255 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216014 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271504 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276634 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265515 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252682 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204416 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231843 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258205 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303943 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611754 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799426 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654629 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667298 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537627 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569642 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620093 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640002 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899568 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907086 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093382 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789002 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812434 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100383 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140726 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09397 s, 11.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8792 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.609 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.602954 s, 27.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636857 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776693 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192338 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146875 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221827 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232114 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237186 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208919 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276922 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222353 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249461 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241644 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248486 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297454 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297217 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287957 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00829284 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461376 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619397 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628255 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614155 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333317 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290401 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335759 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634408 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461765 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410725 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438867 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483161 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598866 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834403 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067417 s, 15.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.45542 s, 14.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94137 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.441157 s, 38.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983682 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946049 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798419 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199419 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189072 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195225 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305617 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233624 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220954 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308602 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336324 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25807 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31096 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336107 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325905 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310871 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112831 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977376 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375455 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093688 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12665 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983512 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121547 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745379 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133308 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729744 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107761 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120403 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811485 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100492 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875629 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108213 s, 9.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8865 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87865 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.894 s, 8.1 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96016 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (451s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 03:57:18 (1790927838) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.519338 s, 32.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656041 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710517 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754192 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858174 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101484 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689263 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078135 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10992 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129854 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145894 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175565 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141614 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155271 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174296 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180092 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238399 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029106 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300969 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307362 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391087 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309936 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307534 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314119 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053715 s, 19.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370936 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103919 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472129 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404963 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546307 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577004 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897743 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849602 s, 12.3 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9362 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98059 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (56s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 03:58:14 (1790927894) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.18@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.18@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.784167 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926449 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729599 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841509 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178592 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175243 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189822 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109243 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105407 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469946 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315992 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318011 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211668 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334835 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186383 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250827 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324916 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210821 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127338 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197986 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16479 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236968 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223402 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172453 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177369 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243543 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194317 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219887 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249064 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259204 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210205 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261671 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225072 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=11 removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87357 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1246 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (56s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 03:59:10 (1790927950) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.18@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.41038 s, 11.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106361 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101475 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108947 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159447 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248656 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280384 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319175 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337678 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276968 s, 3.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d959031d800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364541 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422338 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420235 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400441 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450335 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450521 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480605 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff8d959031d800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124587 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120875 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115212 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933143 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443025 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855891 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108965 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758547 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152677 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148049 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171693 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873104 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140118 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111269 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116347 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115359 s, 9.1 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.955 s, 7.5 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.67068 s, 18.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (59s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 04:00:09 (1790928009) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.772822 s, 21.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984717 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139187 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233261 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134686 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162786 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323449 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334785 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241001 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308653 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173316 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213179 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205218 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245062 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298232 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235015 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223332 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00299159 s, 351 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00988902 s, 106 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00225428 s, 465 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332353 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034167 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00883586 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667689 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807774 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777531 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892696 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080434 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748103 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633292 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158395 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614563 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109992 s, 9.5 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9197 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96344 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.14949 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97816 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (93s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 04:01:42 (1790928102) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.08391 s, 15.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889243 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124674 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18083 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144201 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220729 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167517 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157983 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259318 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27016 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245502 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274534 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301339 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271482 s, 3.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229722 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327261 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354009 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352247 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541357 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218994 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437149 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199919 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195559 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104006 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329144 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900242 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380109 s, 27.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428939 s, 24.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438454 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509438 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540096 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559243 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566497 s, 18.5 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8996 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0009 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (77s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 04:02:59 (1790928179) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.845743 s, 19.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114916 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158761 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152913 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207995 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23981 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327814 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352945 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434038 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433135 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.571117 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.59414 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.56836 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.577849 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.535759 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.607691 s, 1.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.657489 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491789 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111211 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00267128 s, 393 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196425 s, 53.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108301 s, 96.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382238 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00593031 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241151 s, 435 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00244397 s, 429 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121554 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243704 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183092 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214605 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107666 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901865 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224534 s, 46.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9465 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9526 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg318-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.568165 s, 29.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908314 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999319 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17392 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232965 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250428 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236068 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264194 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225748 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298249 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261592 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277906 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285261 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28728 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225027 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280006 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327853 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031567 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298375 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129325 s, 81.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489976 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130157 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00285046 s, 368 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573766 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878353 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888337 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280909 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612482 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617732 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06797 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680287 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120104 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709305 s, 14.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.617 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0444 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7907 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94411 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (146s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 04:05:26 (1790928326) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre opts: Stopping clients: oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg318-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg318-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.18,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg318-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.18,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg318-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.18,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/71262/testresults/sanityn-zfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.18,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg318-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: kmemleak disabled oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Started clients oleg318-client.virtnet: 192.168.203.118@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Started clients oleg318-client.virtnet: 192.168.203.118@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d95859cc000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d9585f6c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d95859cc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d9585f6c800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.54699 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583297 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732342 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126427 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432038 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182249 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210226 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12966 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167253 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101446 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20263 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170815 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15746 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150674 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115781 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084459 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209856 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519383 s, 20.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494343 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038971 s, 26.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438947 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625998 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624465 s, 16.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500589 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654889 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101617 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104755 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715577 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789947 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086199 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100964 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880466 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775538 s, 13.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.31166 s, 12.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08946 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.712345 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111351 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108061 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827719 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104777 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713483 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232572 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21658 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149027 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144284 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106696 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283437 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135088 s, 7.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d95859cc000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19615 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441779 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465555 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362609 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d95859cc000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120137 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730793 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163831 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777254 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767233 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124643 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131427 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128023 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894836 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944712 s, 11.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992855 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10039 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147455 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105813 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153748 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231888 s, 4.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg318-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.50879 s, 12.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93035 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (252s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 04:09:37 (1790928577) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (19s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 04:09:56 (1790928596) On MGS 192.168.203.118, active = nodemap.active=1 On MGS 192.168.203.118, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.653334 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950413 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08266 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124393 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226923 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202537 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244569 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116858 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23845 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287716 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427187 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209069 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395027 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402012 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52772 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347931 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475662 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d95859cc000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121733 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149434 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411161 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753144 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109218 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468648 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052358 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883941 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436835 s, 24.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784892 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625965 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605012 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664495 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740506 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233222 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133063 s, 7.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8639 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3472 s, 19.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.118, TBF.id = On MGS 192.168.203.118, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.787094 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071391 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150735 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157188 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27983 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297164 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222256 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311491 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324625 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280813 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391053 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416914 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37065 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d95859cc000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8d95859cc000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442336 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34984 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420245 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405254 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722734 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451958 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751844 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131646 s, 79.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553752 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870077 s, 12.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861558 s, 12.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303699 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476978 s, 22.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00650653 s, 161 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d95859cc000.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496417 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198443 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179592 s, 58.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416236 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271602 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927112 s, 11.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.91128 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.10067 s, 33.8 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg318-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 On MGS 192.168.203.118, active = nodemap.active=1 On MGS 192.168.203.118, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.678093 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106605 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100177 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124769 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172612 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203502 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203242 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252569 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211104 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234674 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352452 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388123 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325425 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441501 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380389 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d95859cc000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8d95859cc000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456401 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.469435 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147634 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199938 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197436 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176989 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179349 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208727 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215741 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248724 s, 4.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271894 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257672 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23794 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240132 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240787 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248814 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243523 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25438 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d95859cc000.lock_unused_count=1 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96368 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0089 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.118, TBF.id = On MGS 192.168.203.118, active = nodemap.active=0 PASS 77kj (191s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 04:13:07 (1790928787) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 04:13:17 (1790928797) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg318-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.24337 s, 323 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg318-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1172 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (57s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 04:14:14 (1790928854) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.652188 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108711 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866163 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646443 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205786 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761946 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208526 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242242 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386062 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316363 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312021 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424459 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397327 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412249 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472152 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521349 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.510898 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d95859cc000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185905 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935345 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099967 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993116 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594572 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12124 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766995 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812422 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605078 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857386 s, 12.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603502 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130629 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143063 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105879 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13859 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709258 s, 14.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3229 s, 8.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.51819 s, 19.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.443825 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595322 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890982 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661173 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989305 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128501 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219772 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155303 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264706 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261659 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263379 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184352 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159585 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172866 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207907 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224375 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229759 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216397 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790819 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880811 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317127 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561354 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549904 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383417 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451219 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972207 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483365 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668929 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722829 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535621 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555846 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559194 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821223 s, 12.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg318-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg318-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.39608 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17487 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (97s) == sanityn test 77o: Changing rank should not panic ====== 04:15:51 (1790928951) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 04:16:05 (1790928965) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (114s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 04:17:59 (1790929079) mds.MDS.mdt.nrs_policies=tbf oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg318-client: oleg318-server: ssh exited with exit code 36 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg318-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg318-client: oleg318-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (36s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 04:18:34 (1790929114) striped dir -i0 -c1 -H crush2 /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 293548 kill md_thread xargs PASS 77r (55s) == sanityn test 77s: Check TBF LRU shrinker ============== 04:19:29 (1790929169) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (61s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 04:20:30 (1790929230) oleg318-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg318-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg318-client: oleg318-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 04:20:41 (1790929241) striped dir -i1 -c2 -H crush /mnt/lustre/d79.sanityn File: /mnt/lustre/d79.sanityn Size: 32768 Blocks: 16 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129637319639057 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 04:20:43.000000000 -0400 Modify: 2026-10-02 04:20:43.000000000 -0400 Change: 2026-10-02 04:20:43.000000000 -0400 Birth: 2026-10-02 04:20:44.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 04:21:00 (1790929260) total: 10 open/close in 0.20 seconds: 49.48 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 Finish migration, then checking.. PASS 80a (12s) == sanityn test 80b: Accessing directory during migration ========================================================== 04:21:12 (1790929272) start migration thread 300961 accessing the migrating directory for 1 minute... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete concurrent access 236 failures, 340 successes PASS 80b (90s) == sanityn test 81a: rename and stat under striped directory ========================================================== 04:22:42 (1790929362) stat: cannot statx '/mnt/lustre/d81a.sanityn/d0/0': No such file or directory File: /mnt/lustre/d81a.sanityn/d0/0 Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272381367894 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 04:22:45.000000000 -0400 Modify: 2026-10-02 04:22:45.000000000 -0400 Change: 2026-10-02 04:22:46.000000000 -0400 Birth: 2026-10-02 04:22:47.000000000 -0400 PASS 81a (10s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 04:22:52 (1790929372) - open/close 658 (time 1790929387.26 total 10.01 last 65.76) - open/close 1344 (time 1790929397.26 total 20.01 last 68.55) - open/close 1987 (time 1790929407.28 total 30.03 last 64.20) total: 2500 open/close in 37.46 seconds: 66.74 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 303314 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (506s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 04:31:18 (1790929878) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 04:31:22 (1790929882) - open/close 756 (time 1790929897.97 total 10.02 last 75.46) - open/close 1461 (time 1790929907.97 total 20.02 last 70.49) - open/close 2179 (time 1790929917.98 total 30.03 last 71.75) total: 2500 open/close in 34.67 seconds: 72.10 ops/second - open/close 885 (time 1790929936.85 total 10.00 last 88.46) - open/close 1777 (time 1790929946.86 total 20.01 last 89.11) total: 2500 open/close in 27.80 seconds: 89.93 ops/second snapshot_time 1790930102.850996438 secs.nsecs start_time 1790929886.890391797 secs.nsecs elapsed_time 215.960604641 secs.nsecs open 2505 samples [usecs] 132 24500 3636825 11801384841 close 2505 samples [usecs] 138 14549 1040552 1410972002 mknod 2501 samples [usecs] 460 18065 3014513 8126897505 unlink 2500 samples [usecs] 597 167971 5973049 73616179095 mkdir 2 samples [usecs] 2083 27497 29580 760423898 rmdir 2 samples [usecs] 8649 18342 26991 411234165 rename 2500 samples [usecs] 409 74346 17291528 254478300836 getattr 3766 samples [usecs] 1 7189 235624 320991342 statfs 42 samples [usecs] 10 1128 2139 1328505 parallel_rename_file 2500 samples [usecs] 409 74346 17291528 254478300836 crossdir_rename 2500 samples [usecs] 409 74346 17291528 254478300836 readpage 17 samples [usecs] 5594 138071 449884 27646225922 snapshot_time 1790930102.851090806 secs.nsecs start_time 1790929886.890448954 secs.nsecs elapsed_time 215.960641852 secs.nsecs open 2504 samples [usecs] 115 39880 2893193 7315125643 close 2504 samples [usecs] 127 12339 886444 822892332 mknod 2500 samples [usecs] 414 39780 2472733 5974351709 unlink 2500 samples [usecs] 537 41377 5335811 24410638551 mkdir 2 samples [usecs] 922 52324 53246 2738651060 rmdir 2 samples [usecs] 14629 20169 34798 620796202 rename 2500 samples [usecs] 445 51176 13350590 142629197072 getattr 3760 samples [usecs] 10 8023 255995 290688055 statfs 42 samples [usecs] 7 247 1359 121119 parallel_rename_file 2500 samples [usecs] 445 51176 13350590 142629197072 crossdir_rename 2500 samples [usecs] 445 51176 13350590 142629197072 readpage 4 samples [usecs] 6270 75743 117036 6405632354 PASS 81d (225s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 04:35:07 (1790930107) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.8439 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.8439 PASS 82 (9s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 04:35:16 (1790930116) start pid 317380 to create/unlink striped directory start pid 317381 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6204: 317380 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6204: 317381 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (127s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 04:37:23 (1790930243) fail_loc=0x8000060b PASS 84 (12s) == sanityn test 85: Lustre API root cache race =========== 04:37:35 (1790930255) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1790930257 Finishing test test1 at 1790930263 PASS 85 (13s) == sanityn test 90: open/create and unlink striped directory ========================================================== 04:37:48 (1790930268) start pid 322171 to open/create under striped directory start pid 322172 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6271: 322171 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6271: 322172 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (186s) == sanityn test 91: chmod and unlink striped directory === 04:40:54 (1790930454) start pid 324730 to chmod striped directory start pid 324732 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6308: 324730 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6308: 324732 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (187s) == sanityn test 92: create remote directory under orphan directory ========================================================== 04:44:01 (1790930641) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (6s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 04:44:08 (1790930648) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 3607 0xe17 0x2c0000400 0 3611 0xe1b 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4530 0x11b2 0x280000401 1 4493 0x118d 0x2c0000401 rc1=0 and rc2=0 PASS 93 (15s) == sanityn test 94: signal vs CP callback race =========== 04:44:23 (1790930663) osc.lustre-OST0000-osc-ffff8d95859cc000.idle_timeout=0 osc.lustre-OST0000-osc-ffff8d9585f6c800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8d95859cc000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8d9585f6c800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.193404 s, 42.4 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.99619 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6395: 329193 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.044991 s, 91.0 kB/s osc.lustre-OST0000-osc-ffff8d95859cc000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d9585f6c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d95859cc000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d9585f6c800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 04:44:40 (1790930680) llite.lustre-ffff8d95859cc000.fast_read=0 llite.lustre-ffff8d9585f6c800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0680527 s, 30.8 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8d95859cc000.fast_read=1 llite.lustre-ffff8d9585f6c800.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 04:45:00 (1790930700) llite.lustre-ffff8d95859cc000.fast_read=0 llite.lustre-ffff8d9585f6c800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0119448 s, 1.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8d95859cc000.fast_read=1 llite.lustre-ffff8d9585f6c800.fast_read=1 PASS 95b (12s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 04:45:12 (1790930712) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 04:45:15 (1790930715) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00501068 s, 817 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 04:45:22 (1790930722) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118027 s, 17.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (6s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 04:45:28 (1790930728) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155858 s, 13.5 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (6s) == sanityn test 100e: DoM: read on open and file size ==== 04:45:34 (1790930734) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00660863 s, 1.8 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 04:45:41 (1790930741) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00349766 s, 1.2 MB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 04:45:47 (1790930747) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0110434 s, 371 kB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 04:45:53 (1790930753) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00619639 s, 661 kB/s PASS 101c (7s) == sanityn test 102a: Test open by handle of unlinked file ========================================================== 04:46:00 (1790930760) file: /mnt/lustre/f102a.sanityn fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 70 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0x1870:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102a.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: d0 0b 00 40 02 00 00 00 10 14 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd0:0x1410:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102a.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 71 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0x1871:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102a.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: d0 0b 00 40 02 00 00 00 11 14 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd0:0x1411:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102a.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: d0 0b 00 40 02 00 00 00 12 14 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd0:0x1412:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102a (9s) == sanityn test 102b: open by handle of a dir inside an encrypted dir ========================================================== 04:46:09 (1790930769) 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Starting client: oleg318-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.203.118@tcp:/lustre /mnt/lustre2 file: /mnt/lustre/d102b.sanityn/encdir/sub fh_bytes: 32 fh_type: 151 fh_data: a4 13 00 00 02 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a4:0x2:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 PASS 102b (16s) == sanityn test 103: Test size correctness with lockahead ========================================================== 04:46:25 (1790930785) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1790930787 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1790930791 Finishing test test23 at 1790930791 PASS 103 (12s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 04:46:38 (1790930798) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 04:46:41 (1790930801) striped dir -i1 -c2 -H crush2 /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 04:46:59 (1790930819) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 04:47:02 (1790930822) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0127078 s, 80.6 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272448475141 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 04:47:03.000000000 -0400 Modify: 2026-10-02 04:47:03.000000000 -0400 Change: 2026-10-02 04:47:03.000000000 -0400 Birth: 2026-10-02 04:47:04.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272448475141 -rw-r--r-- 1 0 0 1790930824 1790930823 1790930823 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272448475141 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 04:47:03.000000000 -0400 Modify: 2026-10-02 04:47:03.000000000 -0400 Change: 2026-10-02 04:47:03.000000000 -0400 Birth: 2026-10-02 04:47:04.000000000 -0400 1790930823 1024 11 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 04:47:09 (1790930829) PASS 106c (6s) == sanityn test 107a: Basic grouplock conflict =========== 04:47:15 (1790930835) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.232005 s, 45.2 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 4499 0x1193 0x2c0000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 First grouplock blocks second one PASS 107a (8s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 04:47:24 (1790930844) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.272165 s, 38.5 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4539 0x11bb 0x280000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8439 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 04:47:37 (1790930857) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00545531 s, 1.8 MB/s llite.lustre-ffff8d9585f69800.hybrid_io=0 llite.lustre-ffff8d9585f6f800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.75514 s, 1.8 MB/s PASS 108a (11s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 04:47:48 (1790930868) 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: kmemleak disabled oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre 192.168.203.118@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.118@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg318-client.virtnet /mnt/lustre (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: kmemleak disabled oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre2 PASS 109 (61s) == sanityn test 110: do not grant another lock on resend ========================================================== 04:48:50 (1790930930) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272482029571 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 04:48:51.000000000 -0400 Modify: 2026-10-02 04:48:51.000000000 -0400 Change: 2026-10-02 04:48:51.000000000 -0400 Birth: 2026-10-02 04:48:53.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg318-client.virtnet oleg318-client.virtnet PASS 110 (83s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 04:50:12 (1790931012) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 7240: 345712 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (12s) == sanityn test 112: update max-inherit in default LMV === 04:50:24 (1790931024) lmv.lustre-clilmv-ffff8d958332c000.qos_threshold_rr=90 lmv.lustre-clilmv-ffff8d95a1776800.qos_threshold_rr=90 PASS 112 (12s) == sanityn test 113: check servers of specified fs ======= 04:50:36 (1790931036) lustre-OST0000-osc-ffff8d95a1776800 active. lustre-OST0001-osc-ffff8d95a1776800 active. lustre-OST0000-osc-ffff8d95a1776800 active. lustre-OST0001-osc-ffff8d95a1776800 active. MGC192.168.203.118@tcp active. PASS 113 (6s) == sanityn test 114: implicit default LMV inherit ======== 04:50:43 (1790931043) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (24s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 04:51:07 (1790931067) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 04:51:10 (1790931070) llite.lustre-ffff8d958332c000.intent_mkdir=1 llite.lustre-ffff8d95a1776800.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000013a0:0xfd:0x0] 1 [0x240000bd2:0xeb:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none PASS 116 (7s) == sanityn test 121: trunc append race =================== 04:51:17 (1790931077) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.025177 s, 813 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0709894 s, 288 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 04:51:26 (1790931086) PASS 122 (15s) == sanityn test 200: service remains healthy while able to process request ========================================================== 04:51:41 (1790931101) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 striped dir -i0 -c2 -H crush /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1769 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1624 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2013 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2322 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.198 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2234 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2172 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2454 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3234 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3037 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.312 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2806 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3436 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2913 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.321 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4348 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4429 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4073 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg318-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d958332c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d958332c000.ost_server_uuid in IDLE state after 0 sec PASS 200 (80s) cleanup: ====================================================== == sanityn test complete, duration 11481 sec ============= 04:53:03 (1790931183) === sanityn: start cleanup 04:53:05 (1790931185) === Stopping clients: oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 04:57:37 (1790931457) ===