-----============= acceptance-small: sanityn ============----- Fri Sep 4 04:19:40 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg311-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 04:20:02 (1788510002) === Starting client oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Mount client oleg311-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Started clients oleg311-client.virtnet: 192.168.203.111@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg311-client.virtnet: executing check_config_client /mnt/lustre oleg311-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg311-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8efe02fa7000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8efe124ca000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8efe02fa7000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8efe124ca000.idle_timeout=debug disable quota as required oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 04:20:38 (1788510038) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414466 s, 25.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7632] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 04:20:40 (1788510040) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 04:20:48 (1788510048) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 04:20:54 (1788510054) -rw-r--r-- 1 root root 0 Sep 4 04:20 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 04:21:01 (1788510061) -rw-r--r-- 1 root root 0 Sep 4 04:21 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 04:21:07 (1788510067) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 04:21:13 (1788510073) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 04:21:20 (1788510080) total 1 -rwxrwxrwx 1 root root 0 Sep 4 04:20 f2a -rwxrwxrwx 1 root root 0 Sep 4 04:21 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 4 04:20 f2a -rwxrwxrwx 1 root root 0 Sep 4 04:21 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0169538 s, 30.2 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 04:21:27 (1788510087) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 04:21:31 (1788510091) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.332703 s, 6.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (9s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 04:21:39 (1788510099) PASS 3 (8s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 04:21:47 (1788510107) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 04:21:54 (1788510114) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 04:22:01 (1788510121) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 04:22:07 (1788510127) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (5s) == sanityn test 8: remove of open special file on other node ==================================================================== 04:22:12 (1788510132) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 04:22:19 (1788510139) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 04:22:26 (1788510146) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401793 s, 26.1 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0782574 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 04:22:33 (1788510153) 1+0 records in 1+0 records out 1 byte copied, 0.0062151 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0244606 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0175404 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0100336 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0134878 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0114101 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0130114 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0141175 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0217622 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0136692 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0135632 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0178532 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 04:22:42 (1788510162) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0376054 s, 81.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0978855 s, 41.8 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00293827 s, 1.0 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 04:22:49 (1788510169) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 04:22:56 (1788510176) warning: '-runtime' deprecated, use '-t runtime' instead - link 320 (time 1788510188.79 total 10.02 last 31.92) - link 629 (time 1788510198.80 total 20.04 last 30.86) - link 926 (time 1788510208.81 total 30.04 last 29.69) - link 1250 (time 1788510218.83 total 40.07 last 32.32) - link 1536 (time 1788510228.84 total 50.08 last 28.57) - link 1858 (time 1788510238.86 total 60.10 last 32.14) - link 2184 (time 1788510248.89 total 70.12 last 32.52) - link 2495 (time 1788510258.90 total 80.13 last 31.06) - link 2816 (time 1788510268.91 total 90.15 last 32.05) total: 3105 link in 100.02 seconds: 31.04 ops/second using seed 2742771913 running for 100 seconds - stat 4521 (time 1788510190 ; total 11 ; last 11) - stat 10000 (time 1788510200 ; total 21 ; last 10) - stat 14899 (time 1788510211 ; total 32 ; last 11) - stat 19564 (time 1788510222 ; total 43 ; last 11) - stat 20000 (time 1788510222 ; total 43 ; last 0) - stat 23708 (time 1788510233 ; total 54 ; last 11) - stat 26432 (time 1788510244 ; total 65 ; last 11) - stat 29269 (time 1788510255 ; total 76 ; last 11) - stat 30000 (time 1788510258 ; total 79 ; last 3) - stat 32580 (time 1788510269 ; total 90 ; last 11) total: 34448 stats in 100 seconds: 344.480011 stats/second - unlinked 0 (time 1788510229 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3105) error: No such file or directory total: 3104 unlinks in 91 seconds: 34.109890 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21430) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21432) - No such process PASS 12 (150s) == sanityn test 13: test directory page revocation ======= 04:25:26 (1788510326) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 04:25:35 (1788510335) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22616 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 04:25:42 (1788510342) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23196 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 04:25:48 (1788510348) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23774 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 04:25:56 (1788510356) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24356 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 04:26:04 (1788510364) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24938 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 04:26:13 (1788510373) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (53s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 04:27:05 (1788510425) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.578889 s, 14.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.426763 s, 19.7 MB/s Chance of close/open is 1 in 50 Seed set to 431 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1788510432.410111 mapwrite 0x25b7f4 thru 0x25fef5 (0x4702 bytes) truncating to largest ever: 0x80c362 000002[0] 1788510432.644191 trunc from 0x25fef6 to 0x80c361 (0x5ac46c bytes) 000004[1] 1788510432.747627 mapread 0x5c0ef8 thru 0x5cd817 (0xc920 bytes) 000006[1] 1788510432.864780 read 0x3a6552 thru 0x3b0434 (0x9ee3 bytes) 000008[1] 1788510432.931166 mapread 0x015a8a thru 0x01df28 (0x849f bytes) 000010[1] 1788510432.972342 punch from 0x522a70 to 0x52930d (0x689e bytes) 000012[0] 1788510433.025590 write 0x632319 thru 0x63c6e0 (0xa3c8 bytes) 000013[0] 1788510433.077412 punch from 0x351c8f to 0x35a7d2 (0x8b44 bytes) 000014[0] 1788510433.118482 mapread 0x2fafb0 thru 0x301617 (0x6668 bytes) 000015[1] 1788510433.215731 mapwrite 0x39092b thru 0x39f845 (0xef1b bytes) 000018[1] 1788510433.621118 mapread 0x09b9b4 thru 0x0a162d (0x5c7a bytes) 000020[0] 1788510433.746541 mapwrite 0x164f5a thru 0x16bd66 (0x6e0d bytes) 000021[1] 1788510433.967283 punch from 0x02a941 to 0x02d2f5 (0x29b5 bytes) 000023[0] 1788510434.066225 read 0x63ebb5 thru 0x643156 (0x45a2 bytes) 000024[1] 1788510434.093585 mapread 0x130437 thru 0x13664d (0x6217 bytes) 000025[1] 1788510434.132497 mapwrite 0x9b42ae thru 0x9c1e70 (0xdbc3 bytes) 000026[1] 1788510434.345620 mapwrite 0x13bda3 thru 0x140602 (0x4860 bytes) 000027[0] 1788510434.445442 write 0x5ca51f thru 0x5d7135 (0xcc17 bytes) 000029[1] 1788510434.566204 punch from 0x6fe3bb to 0x701130 (0x2d76 bytes) 000033[0] 1788510434.728565 trunc from 0x9c1e71 to 0x61eb07 (0x3a3369 bytes) 000034[1] 1788510434.776991 read 0x1d27e5 thru 0x1d7fce (0x57ea bytes) 000036[0] 1788510434.817487 read 0x51a1db thru 0x51ca38 (0x285e bytes) 000038[0] 1788510434.848687 mapwrite 0x08490a thru 0x086f13 (0x260a bytes) 000039[0] 1788510435.019196 mapread 0x235b34 thru 0x239a94 (0x3f61 bytes) 000041[1] 1788510435.080078 write 0x772ea9 thru 0x77dfbe (0xb116 bytes) 000042[0] 1788510435.129455 write 0x924976 thru 0x92c414 (0x7a9f bytes) 000044[1] 1788510435.208513 mapwrite 0x9fdcd1 thru 0x9fffff (0x232f bytes) 000046[0] 1788510435.430806 punch from 0x127cb8 to 0x12f3d9 (0x7722 bytes) 000050[0] 1788510435.523641 read 0x80acea thru 0x80bcf0 (0x1007 bytes) 000052[0] 1788510435.580262 punch from 0x4f1624 to 0x501243 (0xfc20 bytes) 000054[1] 1788510435.648110 mapread 0x032ae5 thru 0x03f252 (0xc76e bytes) 000055[0] 1788510435.752937 read 0x8304a2 thru 0x83a676 (0xa1d5 bytes) 000056[0] 1788510435.812110 mapread 0x7ccd32 thru 0x7cdc45 (0xf14 bytes) 000057[0] 1788510435.851806 mapread 0x2d14ce thru 0x2d1cea (0x81d bytes) 000058[1] 1788510435.906551 mapread 0x99d5e9 thru 0x9a9f37 (0xc94f bytes) 000059[0] 1788510435.991328 write 0x056593 thru 0x065935 (0xf3a3 bytes) 000061[1] 1788510436.040901 trunc from 0xa00000 to 0x796d8e (0x269271 bytes) 000062[1] 1788510436.145239 trunc from 0x796d8f to 0x05ef10 (0x737e7e bytes) 000063[1] 1788510436.215651 read 0x00dd4a thru 0x01859f (0xa856 bytes) truncating to largest ever: 0x9dd6d4 000065[0] 1788510436.298343 trunc from 0x05ef11 to 0x9dd6d3 (0x97e7c3 bytes) 000066[0] 1788510436.463780 trunc from 0x9dd6d4 to 0x572007 (0x46b6cc bytes) 000067[1] 1788510436.569027 trunc from 0x572008 to 0x639a67 (0xc7a60 bytes) 000068[0] 1788510436.630215 punch from 0x62dd63 to 0x63732e (0x95cc bytes) 000069[1] 1788510436.681528 read 0x35969e thru 0x36003b (0x699e bytes) 000071[0] 1788510436.732165 write 0x782d5c thru 0x7849ad (0x1c52 bytes) 000074[1] 1788510436.794768 mapread 0x1c81e9 thru 0x1ccffc (0x4e14 bytes) 000076[0] 1788510436.853718 punch from 0x4a95cd to 0x4b6d19 (0xd74d bytes) 000078[1] 1788510436.896959 trunc from 0x7849ae to 0x43b5c2 (0x3493eb bytes) 000079[0] 1788510436.951293 write 0x7aa13f thru 0x7b07d2 (0x6694 bytes) 000080[1] 1788510437.006728 trunc from 0x7b07d3 to 0x0ca4c9 (0x6e6309 bytes) 000081[1] 1788510437.125938 trunc from 0x0ca4ca to 0x4c9bbe (0x3ff6f5 bytes) 000082[1] 1788510437.218236 mapread 0x19866a thru 0x1a015e (0x7af5 bytes) 000083[1] 1788510437.239487 mapread 0x265d2e thru 0x26e32a (0x85fd bytes) 000084[0] 1788510437.350125 read 0x2230d4 thru 0x227385 (0x42b2 bytes) 000086[0] 1788510437.395291 mapread 0x127933 thru 0x131e7c (0xa54a bytes) 000087[1] 1788510437.451266 trunc from 0x4c9bbf to 0x88eefe (0x3c5340 bytes) 000089[1] 1788510437.542179 read 0x32b409 thru 0x337dfe (0xc9f6 bytes) 000090[1] 1788510437.612645 trunc from 0x88eeff to 0x5a867e (0x2e6880 bytes) 000093[0] 1788510437.711808 read 0x0ae7d9 thru 0x0ba76d (0xbf95 bytes) 000094[0] 1788510437.775693 read 0x39743a thru 0x39fdde (0x89a5 bytes) 000096[0] 1788510437.832004 mapread 0x256f14 thru 0x25fac6 (0x8bb3 bytes) 000097[0] 1788510437.899105 punch from 0x235d7e to 0x2431c4 (0xd447 bytes) 000098[1] 1788510437.941744 trunc from 0x5a867f to 0x39a6ee (0x20df90 bytes) 000099[0] 1788510437.992330 write 0x954441 thru 0x95467f (0x23f bytes) 000100[0] 1788510438.035393 read 0x0a9b0b thru 0x0b9251 (0xf747 bytes) 000102[1] 1788510438.121629 read 0x718524 thru 0x71a7d3 (0x22b0 bytes) 000103[1] 1788510438.179347 read 0x87fd99 thru 0x8844f2 (0x475a bytes) 000104[1] 1788510438.230933 write 0x3094fa thru 0x319178 (0xfc7f bytes) 000105[1] 1788510438.284364 mapwrite 0x9b7ba1 thru 0x9c6851 (0xecb1 bytes) 000106[1] 1788510438.640932 trunc from 0x9c6852 to 0x05873f (0x96e112 bytes) 000107[1] 1788510438.681272 read 0x053b0e thru 0x05873f (0x4c32 bytes) 000108[0] 1788510438.715187 write 0x6b9ae8 thru 0x6bd6d4 (0x3bed bytes) 000109[1] 1788510438.780289 mapwrite 0x0a1651 thru 0x0aaa95 (0x9445 bytes) 000111[0] 1788510439.263382 read 0x14c2d5 thru 0x15396a (0x7696 bytes) 000114[0] 1788510439.430072 write 0x4f2142 thru 0x4f61c2 (0x4081 bytes) 000116[1] 1788510439.484482 trunc from 0x6bd6d5 to 0x4c5cf3 (0x1f79e1 bytes) 000117[1] 1788510439.603196 mapwrite 0x8b329d thru 0x8b4b4c (0x18b0 bytes) 000118[0] 1788510439.917251 mapwrite 0x087d4b thru 0x088abd (0xd73 bytes) 000120[1] 1788510440.173581 read 0x8784a3 thru 0x87c130 (0x3c8e bytes) 000121[1] 1788510440.239699 mapwrite 0x4835d1 thru 0x48f2ea (0xbd1a bytes) 000122[0] 1788510440.504817 punch from 0x289309 to 0x28ff8e (0x6c86 bytes) 000123[0] 1788510440.574228 mapread 0x4fa951 thru 0x50142e (0x6ade bytes) 000124[1] 1788510440.671227 mapread 0x4f597b thru 0x4ff3fe (0x9a84 bytes) 000128[0] 1788510440.768813 write 0x2c80c6 thru 0x2d4986 (0xc8c1 bytes) 000129[1] 1788510440.796560 trunc from 0x8b4b4d to 0x564454 (0x3506f8 bytes) 000130[1] 1788510440.916952 trunc from 0x564455 to 0x90d0b0 (0x3a8c5c bytes) 000134[1] 1788510441.012165 read 0x352a72 thru 0x3600a0 (0xd62f bytes) 000135[0] 1788510441.074346 punch from 0x48b266 to 0x493120 (0x7ebb bytes) 000137[1] 1788510441.212635 mapread 0x8d6777 thru 0x8e1c56 (0xb4e0 bytes) 000138[1] 1788510441.321171 punch from 0x19c3e2 to 0x1a38b1 (0x74d0 bytes) 000139[1] 1788510441.397322 mapwrite 0x2ced83 thru 0x2d3d31 (0x4faf bytes) 000142[0] 1788510441.691349 punch from 0x7d6790 to 0x7e105c (0xa8cd bytes) 000145[0] 1788510441.818086 mapread 0x63481e thru 0x635626 (0xe09 bytes) 000146[1] 1788510441.873051 trunc from 0x90d0b1 to 0x4020d0 (0x50afe0 bytes) 000148[1] 1788510441.949040 punch from 0x1c5150 to 0x1d1185 (0xc036 bytes) 000149[0] 1788510441.979487 punch from 0x0e9cc2 to 0x0f3302 (0x9641 bytes) 000151[0] 1788510442.044497 mapwrite 0x5078b6 thru 0x50d5b6 (0x5d01 bytes) 000153[0] 1788510442.207568 punch from 0x2584d9 to 0x2674c9 (0xeff1 bytes) 000154[1] 1788510442.254216 trunc from 0x50d5b7 to 0x322884 (0x1ead32 bytes) 000155[1] 1788510442.349946 mapread 0x03cbf1 thru 0x04c52e (0xf93e bytes) 000157[1] 1788510442.433926 trunc from 0x322885 to 0x4b745d (0x194bd9 bytes) 000158[1] 1788510442.518784 mapwrite 0x746f13 thru 0x756e5d (0xff4b bytes) 000160[0] 1788510442.763443 punch from 0x2daf6d to 0x2df8e8 (0x497c bytes) 000161[1] 1788510442.794311 mapwrite 0x85f8e5 thru 0x86b0d9 (0xb7f5 bytes) 000162[1] 1788510442.915735 trunc from 0x86b0da to 0x6e93ed (0x181cec bytes) 000163[0] 1788510442.978943 read 0x409384 thru 0x417175 (0xddf2 bytes) 000165[1] 1788510443.064984 mapwrite 0x8f364a thru 0x8fd103 (0x9aba bytes) 000167[1] 1788510443.366419 write 0x945e83 thru 0x94e184 (0x8302 bytes) 000168[1] 1788510443.409078 mapread 0x94d5f7 thru 0x94e184 (0xb8e bytes) 000169[0] 1788510443.432999 mapread 0x907072 thru 0x9150ce (0xe05d bytes) 000170[1] 1788510443.558640 trunc from 0x94e185 to 0x83c9b4 (0x1117d0 bytes) 000172[1] 1788510443.669398 read 0x717e04 thru 0x7252e4 (0xd4e1 bytes) 000174[1] 1788510443.736921 punch from 0x43148f to 0x4331f3 (0x1d65 bytes) 000175[1] 1788510443.788951 mapread 0x6657cb thru 0x6658e3 (0x119 bytes) 000176[1] 1788510443.830787 punch from 0x4f4711 to 0x4fcf12 (0x8802 bytes) 000180[1] 1788510443.941639 trunc from 0x83c9b5 to 0x5a1948 (0x29b06c bytes) 000181[0] 1788510444.025466 trunc from 0x5a1949 to 0x23e795 (0x3631b3 bytes) 000182[1] 1788510444.085509 mapwrite 0x05d68b thru 0x065c0d (0x8583 bytes) 000183[0] 1788510444.281214 trunc from 0x23e796 to 0x6afa84 (0x4712ef bytes) 000184[0] 1788510444.394388 punch from 0x13aa54 to 0x141d12 (0x72bf bytes) 000190[0] 1788510444.520403 trunc from 0x6afa85 to 0x9aa82d (0x2fada9 bytes) 000191[0] 1788510444.552386 mapwrite 0x935744 thru 0x93b182 (0x5a3f bytes) 000192[0] 1788510444.655661 write 0x22a33a thru 0x22f3b2 (0x5079 bytes) 000195[1] 1788510444.727916 write 0x06865f thru 0x07309c (0xaa3e bytes) 000198[0] 1788510444.804453 punch from 0x829875 to 0x8345ce (0xad5a bytes) 000199[0] 1788510444.833783 write 0x436304 thru 0x4372ef (0xfec bytes) 000201[0] 1788510444.857314 read 0x2baf73 thru 0x2bd005 (0x2093 bytes) 000203[0] 1788510444.973578 read 0x2f127e thru 0x2f2785 (0x1508 bytes) 000204[1] 1788510444.988660 mapread 0x6dd526 thru 0x6e0529 (0x3004 bytes) 000206[1] 1788510445.113322 trunc from 0x9aa82e to 0x5b1d3d (0x3f8af0 bytes) 000209[0] 1788510445.239028 mapread 0x318f18 thru 0x324584 (0xb66d bytes) 000210[1] 1788510445.295244 punch from 0x1caaa4 to 0x1d0fdc (0x6539 bytes) 000211[0] 1788510445.333316 mapwrite 0x2100d1 thru 0x21123c (0x116c bytes) 000212[1] 1788510445.425086 mapwrite 0x37882d thru 0x37b357 (0x2b2b bytes) 000214[0] 1788510445.576694 trunc from 0x5b1d3e to 0x4024ef (0x1af84e bytes) 000215[1] 1788510445.718103 mapread 0x253c14 thru 0x258586 (0x4973 bytes) 000218[0] 1788510445.821560 mapread 0x09bc38 thru 0x09d77c (0x1b45 bytes) 000221[0] 1788510445.869421 mapwrite 0x3868e1 thru 0x390f22 (0xa642 bytes) truncating to largest ever: 0x9de0ee 000222[1] 1788510445.944446 trunc from 0x4024f0 to 0x9de0ed (0x5dbbfe bytes) 000224[1] 1788510445.998581 read 0x7012c6 thru 0x70cac8 (0xb803 bytes) 000227[1] 1788510446.151710 mapread 0x8991a2 thru 0x89d4c2 (0x4321 bytes) 000230[0] 1788510446.205820 mapwrite 0x1089ac thru 0x116d47 (0xe39c bytes) 000231[0] 1788510446.374359 mapwrite 0x1cf18d thru 0x1d6133 (0x6fa7 bytes) 000232[0] 1788510446.484777 read 0x461e75 thru 0x466e7b (0x5007 bytes) 000235[1] 1788510446.565643 mapread 0x0f4378 thru 0x0f9e6a (0x5af3 bytes) 000236[1] 1788510446.636527 read 0x7c9981 thru 0x7d1996 (0x8016 bytes) 000237[0] 1788510446.651183 write 0x188d4a thru 0x18f136 (0x63ed bytes) 000239[1] 1788510446.681471 write 0x9ac8cf thru 0x9bbe41 (0xf573 bytes) 000241[1] 1788510446.746860 punch from 0x6b5fd2 to 0x6b9971 (0x39a0 bytes) 000243[1] 1788510446.821695 read 0x85e001 thru 0x864e8c (0x6e8c bytes) 000244[0] 1788510446.844600 punch from 0x533434 to 0x5385eb (0x51b8 bytes) 000246[0] 1788510446.899123 write 0x140aec thru 0x147a35 (0x6f4a bytes) 000247[1] 1788510446.971559 mapwrite 0x881153 thru 0x8854fa (0x43a8 bytes) 000249[0] 1788510447.242822 punch from 0x9a8fd3 to 0x9ac2d0 (0x32fe bytes) 000250[0] 1788510447.257566 write 0x653fd8 thru 0x65a885 (0x68ae bytes) 000251[0] 1788510447.321672 mapread 0x452d89 thru 0x45d277 (0xa4ef bytes) 000252[1] 1788510447.381787 mapwrite 0x70acea thru 0x713c5d (0x8f74 bytes) 000253[1] 1788510447.511614 read 0x56b663 thru 0x5793b3 (0xdd51 bytes) 000256[1] 1788510447.568621 write 0x144d73 thru 0x14baf0 (0x6d7e bytes) 000258[1] 1788510447.588278 punch from 0x5e79eb to 0x5f153f (0x9b55 bytes) 000260[0] 1788510447.595992 trunc from 0x9de0ee to 0x830efe (0x1ad1ef bytes) 000264[1] 1788510447.680136 mapread 0x4a377d thru 0x4a55ea (0x1e6e bytes) 000265[0] 1788510447.743625 punch from 0x43479b to 0x43e441 (0x9ca7 bytes) 000266[1] 1788510447.802120 write 0x5afadf thru 0x5b8d4b (0x926d bytes) 000269[1] 1788510447.949280 mapwrite 0x21eb4a thru 0x226e06 (0x82bd bytes) 000271[0] 1788510448.116535 write 0x5a087a thru 0x5ad76d (0xcef4 bytes) 000273[1] 1788510448.234176 write 0x79782d thru 0x79dfa0 (0x6774 bytes) 000275[0] 1788510448.340222 mapread 0x3f0777 thru 0x3f7d33 (0x75bd bytes) 000278[0] 1788510448.498341 punch from 0x615a12 to 0x619df6 (0x43e5 bytes) 000279[0] 1788510448.530076 trunc from 0x830eff to 0x8f53e6 (0xc44e8 bytes) 000281[0] 1788510448.565392 mapwrite 0x1aaddc thru 0x1afad4 (0x4cf9 bytes) 000283[0] 1788510448.674602 punch from 0x1924a9 to 0x196976 (0x44ce bytes) 000284[0] 1788510448.701358 trunc from 0x8f53e7 to 0x64134d (0x2b4099 bytes) 000285[1] 1788510448.747580 mapwrite 0x542984 thru 0x550cee (0xe36b bytes) 000287[0] 1788510448.963902 mapwrite 0x1caf21 thru 0x1cc727 (0x1807 bytes) 000288[1] 1788510449.018760 punch from 0x2eafa6 to 0x2f91e6 (0xe241 bytes) 000289[1] 1788510449.036947 mapwrite 0x27b6f7 thru 0x27df22 (0x282c bytes) 000290[0] 1788510449.113190 read 0x0a9701 thru 0x0acf1c (0x381c bytes) 000292[0] 1788510449.178311 mapread 0x077694 thru 0x07dc0b (0x6578 bytes) 000296[1] 1788510449.347454 trunc from 0x64134e to 0x4bd0c0 (0x18428d bytes) 000297[1] 1788510449.409178 mapread 0x401774 thru 0x408312 (0x6b9f bytes) 000301[0] 1788510449.544874 punch from 0x1822ec to 0x18a852 (0x8567 bytes) 000304[0] 1788510449.624723 write 0x80025d thru 0x8093e5 (0x9189 bytes) 000305[0] 1788510449.662669 read 0x07b7e6 thru 0x085028 (0x9843 bytes) 000306[1] 1788510449.712176 trunc from 0x8093e6 to 0x2af415 (0x559fd0 bytes) 000307[1] 1788510449.881132 mapread 0x015c61 thru 0x017f3e (0x22de bytes) 000308[1] 1788510449.945671 trunc from 0x2af416 to 0x720fcc (0x471bb7 bytes) 000309[0] 1788510449.978613 write 0x0df305 thru 0x0eb85c (0xc558 bytes) 000311[1] 1788510450.033888 mapread 0x330997 thru 0x33afa6 (0xa610 bytes) 000315[0] 1788510450.137167 write 0x8be05e thru 0x8c669f (0x8642 bytes) 000316[1] 1788510450.170271 read 0x8921e2 thru 0x89947b (0x729a bytes) 000323[1] 1788510450.338925 write 0x777498 thru 0x77a072 (0x2bdb bytes) 000324[1] 1788510450.366557 trunc from 0x8c66a0 to 0x849605 (0x7d09a bytes) 000327[0] 1788510450.413444 mapwrite 0x66ff58 thru 0x675618 (0x56c1 bytes) 000328[0] 1788510450.588964 mapwrite 0x2b2208 thru 0x2b6754 (0x454d bytes) 000330[1] 1788510450.813519 read 0x20584f thru 0x208fd9 (0x378b bytes) 000331[1] 1788510450.841246 mapwrite 0x7136da thru 0x7148d1 (0x11f8 bytes) 000332[1] 1788510451.035536 read 0x6f7646 thru 0x6ff632 (0x7fed bytes) 000333[1] 1788510451.085611 read 0x1e2e6d thru 0x1eb556 (0x86ea bytes) 000334[1] 1788510451.136126 trunc from 0x849606 to 0x0acfd9 (0x79c62c bytes) 000337[1] 1788510451.309984 trunc from 0x0acfda to 0x5fd12a (0x550151 bytes) 000339[1] 1788510451.388319 punch from 0x1c1b43 to 0x1c9558 (0x7a16 bytes) 000344[1] 1788510451.526209 write 0x711c14 thru 0x721240 (0xf62d bytes) 000346[0] 1788510451.531824 trunc from 0x721241 to 0x7e71c7 (0xc5f87 bytes) 000349[1] 1788510451.664631 trunc from 0x7e71c8 to 0x9300d7 (0x148f10 bytes) 000350[0] 1788510451.776946 punch from 0x4445b5 to 0x453738 (0xf184 bytes) 000352[0] 1788510451.823751 mapwrite 0x1d6457 thru 0x1e1595 (0xb13f bytes) 000353[1] 1788510452.007416 trunc from 0x9300d8 to 0x7cdd47 (0x162390 bytes) 000355[0] 1788510452.150441 read 0x28426a thru 0x28ed9d (0xab34 bytes) 000356[1] 1788510452.195708 mapwrite 0x7f7448 thru 0x7fa7f1 (0x33aa bytes) 000358[0] 1788510452.379201 trunc from 0x7fa7f2 to 0x3d16cd (0x429124 bytes) 000360[1] 1788510452.464178 trunc from 0x3d16ce to 0x9aa23a (0x5d8b6d bytes) 000361[1] 1788510452.539821 write 0x444106 thru 0x44cd19 (0x8c14 bytes) 000362[1] 1788510452.572669 write 0x3e93e1 thru 0x3ea092 (0xcb2 bytes) 000365[0] 1788510452.662609 mapwrite 0x80a300 thru 0x81405e (0x9d5f bytes) 000367[0] 1788510452.829232 read 0x443651 thru 0x44b2b4 (0x7c64 bytes) 000369[1] 1788510452.876691 read 0x92aa28 thru 0x939635 (0xec0e bytes) 000370[1] 1788510452.918811 mapwrite 0x5d2699 thru 0x5dea06 (0xc36e bytes) 000371[0] 1788510453.104593 punch from 0x5b0c89 to 0x5b4df1 (0x4169 bytes) 000373[0] 1788510453.185015 mapread 0x465991 thru 0x46b1b3 (0x5823 bytes) 000375[0] 1788510453.262225 mapread 0x842eb2 thru 0x846d33 (0x3e82 bytes) 000382[1] 1788510453.381346 write 0x1638f2 thru 0x169007 (0x5716 bytes) 000383[0] 1788510453.468330 mapwrite 0x0b8e57 thru 0x0c2c47 (0x9df1 bytes) 000385[1] 1788510453.760228 read 0x140569 thru 0x14fce4 (0xf77c bytes) 000387[1] 1788510453.819152 read 0x33eda3 thru 0x34df87 (0xf1e5 bytes) 000389[0] 1788510453.863227 trunc from 0x9aa23b to 0x4b0d6f (0x4f94cb bytes) 000390[0] 1788510453.901210 read 0x2876b4 thru 0x28fed8 (0x8825 bytes) 000393[0] 1788510453.961586 trunc from 0x4b0d70 to 0x0bfc76 (0x3f10f9 bytes) 000394[0] 1788510453.992391 mapread 0x046246 thru 0x04fc4c (0x9a07 bytes) 000395[1] 1788510454.067121 read 0x058846 thru 0x0661da (0xd995 bytes) 000397[1] 1788510454.115158 trunc from 0x0bfc77 to 0x754017 (0x6943a1 bytes) 000398[1] 1788510454.206062 mapread 0x21c3c0 thru 0x21fa00 (0x3641 bytes) 000399[0] 1788510454.326820 write 0x647b28 thru 0x64e18e (0x6667 bytes) 000401[1] 1788510454.401443 read 0x02cfa5 thru 0x033fca (0x7026 bytes) 000402[1] 1788510454.446259 punch from 0x21fe7b to 0x22a2c3 (0xa449 bytes) 000403[1] 1788510454.473179 mapread 0x477c70 thru 0x47ea28 (0x6db9 bytes) 000406[0] 1788510454.612697 punch from 0x4e8052 to 0x4f4dd3 (0xcd82 bytes) 000407[1] 1788510454.649438 mapread 0x276057 thru 0x27720a (0x11b4 bytes) 000408[1] 1788510454.671083 punch from 0x12143c to 0x12c458 (0xb01d bytes) 000409[1] 1788510454.691510 punch from 0x07c1ad to 0x089fb5 (0xde09 bytes) 000413[0] 1788510454.749361 write 0x31f58d thru 0x3252a3 (0x5d17 bytes) 000414[0] 1788510454.776527 trunc from 0x754018 to 0x4ff8b2 (0x254765 bytes) 000418[0] 1788510454.863141 write 0x51efd0 thru 0x52616a (0x719b bytes) 000423[0] 1788510454.930415 punch from 0x4ef2be to 0x4f7263 (0x7fa6 bytes) 000424[1] 1788510454.953346 write 0x99c58f thru 0x9a2cc0 (0x6732 bytes) 000425[1] 1788510454.994115 write 0x1174fe thru 0x121d24 (0xa827 bytes) 000427[0] 1788510455.029813 trunc from 0x9a2cc1 to 0x0242f9 (0x97e9c7 bytes) 000429[1] 1788510455.101885 mapwrite 0x7d3579 thru 0x7d8a47 (0x54cf bytes) 000431[1] 1788510455.314091 write 0x89516d thru 0x8a1e51 (0xcce5 bytes) 000432[0] 1788510455.318649 mapwrite 0x5e75dd thru 0x5edd6f (0x6793 bytes) 000433[0] 1788510455.446806 mapwrite 0x1c14bc thru 0x1c2069 (0xbae bytes) 000434[0] 1788510455.584947 read 0x15433c thru 0x159010 (0x4cd5 bytes) 000435[0] 1788510455.689976 read 0x69fe0d thru 0x6a9558 (0x974c bytes) 000438[0] 1788510455.802531 read 0x6d25c4 thru 0x6e08da (0xe317 bytes) 000439[1] 1788510455.815449 mapwrite 0x198b94 thru 0x1a3be7 (0xb054 bytes) 000440[0] 1788510455.944989 trunc from 0x8a1e52 to 0x2f3eff (0x5adf52 bytes) 000441[0] 1788510455.982152 mapwrite 0x926064 thru 0x932f0d (0xceaa bytes) 000442[0] 1788510456.068977 write 0x35f8ec thru 0x3696f0 (0x9e05 bytes) 000443[0] 1788510456.079651 mapread 0x079930 thru 0x089010 (0xf6e1 bytes) 000444[0] 1788510456.111298 write 0x060b80 thru 0x06d15c (0xc5dd bytes) 000445[0] 1788510456.123806 mapwrite 0x5465c7 thru 0x54767e (0x10b8 bytes) truncating to largest ever: 0x9f0de0 000446[0] 1788510456.188182 trunc from 0x932f0e to 0x9f0ddf (0xbded2 bytes) 000447[1] 1788510456.218045 mapwrite 0x0227d0 thru 0x027b05 (0x5336 bytes) 000448[0] 1788510456.322260 write 0x95d7ef thru 0x962723 (0x4f35 bytes) 000450[0] 1788510456.386870 trunc from 0x9f0de0 to 0x734a44 (0x2bc39b bytes) truncating to largest ever: 0x9fe86a 000451[0] 1788510456.410499 trunc from 0x734a45 to 0x9fe869 (0x2c9e25 bytes) 000456[1] 1788510456.527503 read 0x5321be thru 0x53eea9 (0xccec bytes) 000458[0] 1788510456.636948 mapwrite 0x141f0d thru 0x144d38 (0x2e2c bytes) 000461[1] 1788510456.848383 read 0x452ce1 thru 0x4618fd (0xec1d bytes) 000462[0] 1788510456.875517 trunc from 0x9fe86a to 0x296732 (0x768137 bytes) 000465[0] 1788510457.005741 trunc from 0x296733 to 0x1e62a1 (0xb0491 bytes) 000466[1] 1788510457.102740 mapwrite 0x5d86e8 thru 0x5e3576 (0xae8f bytes) 000469[1] 1788510457.382235 mapread 0x4f9ced thru 0x4fe43d (0x4751 bytes) 000471[0] 1788510457.421411 punch from 0x00a48f to 0x017fa8 (0xdb1a bytes) 000473[0] 1788510457.484384 mapread 0x587647 thru 0x58b0a7 (0x3a61 bytes) 000474[1] 1788510457.570466 trunc from 0x5e3577 to 0x974904 (0x39138e bytes) 000476[1] 1788510457.652726 trunc from 0x974905 to 0x2b7acf (0x6bce35 bytes) 000477[0] 1788510457.702366 trunc from 0x2b7ad0 to 0x1acacf (0x10b000 bytes) 000480[1] 1788510457.794951 trunc from 0x1acad0 to 0x585c70 (0x3d91a1 bytes) 000482[0] 1788510457.885791 write 0x44299b thru 0x44cea3 (0xa509 bytes) 000483[0] 1788510457.926447 mapread 0x463b31 thru 0x46be74 (0x8344 bytes) 000485[1] 1788510457.978920 trunc from 0x585c71 to 0x5da337 (0x546c7 bytes) 000487[0] 1788510458.024353 punch from 0x5c9e42 to 0x5d2455 (0x8614 bytes) 000488[0] 1788510458.061812 mapread 0x2641ff thru 0x2678cf (0x36d1 bytes) 000489[0] 1788510458.137074 trunc from 0x5da338 to 0x1cd2f3 (0x40d044 bytes) 000490[0] 1788510458.169539 punch from 0x1af71b to 0x1bd7c8 (0xe0ae bytes) 000492[0] 1788510458.252083 punch from 0x052301 to 0x05c7d6 (0xa4d6 bytes) 000493[1] 1788510458.306478 punch from 0x1c00f9 to 0x1c87fc (0x8704 bytes) 000495[0] 1788510458.373721 write 0x076cb6 thru 0x077aac (0xdf7 bytes) 000497[1] 1788510458.425335 punch from 0x1889c9 to 0x189eb4 (0x14ec bytes) 000499[1] 1788510458.465777 trunc from 0x1cd2f4 to 0x4f10df (0x323dec bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 458 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000005[1] 1788510459.534238 write 0x84f000 thru 0x85ca22 (0xda23 bytes) 000006[1] 1788510459.578710 write 0x326000 thru 0x32ffd3 (0x9fd4 bytes) 000007[1] 1788510459.629082 write 0x39e000 thru 0x3ac0b6 (0xe0b7 bytes) truncating to largest ever: 0x76843 000009[1] 1788510459.712006 trunc from 0x85ca23 to 0x076842 (0x7e61e0 bytes) 000017[1] 1788510459.836972 mapwrite 0x256000 thru 0x25bf2a (0x5f2b bytes) 000020[1] 1788510460.060083 write 0x358000 thru 0x35928d (0x128e bytes) truncating to largest ever: 0x5490cd 000023[0] 1788510460.083432 trunc from 0x35928e to 0x5490cc (0x1efe3f bytes) 000025[1] 1788510460.169825 read 0x34c000 thru 0x34ed65 (0x2d66 bytes) 000027[0] 1788510460.209218 mapwrite 0x071000 thru 0x0760c2 (0x50c3 bytes) 000028[1] 1788510460.351914 punch from 0x38e774 to 0x39480a (0x6097 bytes) 000030[1] 1788510460.398171 read 0x26b000 thru 0x27212e (0x712f bytes) 000031[0] 1788510460.408997 write 0x00c000 thru 0x012ea6 (0x6ea7 bytes) 000032[1] 1788510460.438601 write 0x21f000 thru 0x225c9d (0x6c9e bytes) 000035[0] 1788510460.514356 mapread 0x4e7000 thru 0x4f6e8a (0xfe8b bytes) 000036[1] 1788510460.580195 write 0x541000 thru 0x54257f (0x1580 bytes) 000037[0] 1788510460.598656 read 0x3f0000 thru 0x3f5ec6 (0x5ec7 bytes) 000038[0] 1788510460.645993 punch from 0x285fd0 to 0x28c71e (0x674f bytes) 000041[1] 1788510460.695444 mapwrite 0x5ae000 thru 0x5bca35 (0xea36 bytes) 000043[1] 1788510460.841807 read 0x36f000 thru 0x377a8c (0x8a8d bytes) 000044[1] 1788510460.867958 write 0x2cf000 thru 0x2d2ff2 (0x3ff3 bytes) 000045[0] 1788510460.875184 write 0x827000 thru 0x82fa20 (0x8a21 bytes) 000046[1] 1788510460.899052 write 0x362000 thru 0x36539a (0x339b bytes) 000047[1] 1788510460.944616 mapwrite 0x606000 thru 0x615bd2 (0xfbd3 bytes) 000048[1] 1788510461.155617 trunc from 0x82fa21 to 0x0df200 (0x750820 bytes) 000050[1] 1788510461.253601 read 0x076000 thru 0x07e438 (0x8439 bytes) truncating to largest ever: 0x88c69e 000051[0] 1788510461.308875 trunc from 0x0df201 to 0x88c69d (0x7ad49d bytes) 000052[1] 1788510461.377708 write 0x24a000 thru 0x24dc1c (0x3c1d bytes) 000053[0] 1788510461.480269 punch from 0x0cc528 to 0x0d2ec5 (0x699e bytes) 000055[1] 1788510461.512604 write 0x064000 thru 0x06df2c (0x9f2d bytes) 000057[0] 1788510461.608148 write 0x419000 thru 0x41a9bc (0x19bd bytes) 000058[0] 1788510461.653202 mapread 0x130000 thru 0x13b7f1 (0xb7f2 bytes) 000060[1] 1788510461.703364 mapread 0x19c000 thru 0x1a6e97 (0xae98 bytes) 000061[1] 1788510461.775109 mapread 0x59c000 thru 0x5a8464 (0xc465 bytes) 000064[1] 1788510461.851626 trunc from 0x88c69e to 0x4a7b22 (0x3e4b7b bytes) truncating to largest ever: 0x89fec5 000065[1] 1788510461.895661 trunc from 0x4a7b23 to 0x89fec4 (0x3f83a2 bytes) 000069[0] 1788510461.979375 write 0x475000 thru 0x47c874 (0x7875 bytes) 000070[0] 1788510462.028046 mapread 0x645000 thru 0x6483e3 (0x33e4 bytes) 000071[1] 1788510462.047271 punch from 0x861cba to 0x865328 (0x366f bytes) 000072[1] 1788510462.072187 write 0x963000 thru 0x96ef88 (0xbf89 bytes) 000074[0] 1788510462.111164 punch from 0x6c385b to 0x6c79b2 (0x4158 bytes) 000075[0] 1788510462.136970 mapread 0x966000 thru 0x9667bc (0x7bd bytes) 000076[1] 1788510462.173245 mapwrite 0x2c1000 thru 0x2c6ebb (0x5ebc bytes) 000078[0] 1788510462.368296 write 0x36d000 thru 0x37384a (0x684b bytes) 000079[1] 1788510462.392211 write 0x4a1000 thru 0x4a3d10 (0x2d11 bytes) 000080[0] 1788510462.437332 read 0x6cb000 thru 0x6cb3e9 (0x3ea bytes) truncating to largest ever: 0x95598f 000081[1] 1788510462.462920 trunc from 0x96ef89 to 0x95598e (0x195fa bytes) 000083[1] 1788510462.503381 write 0x943000 thru 0x948f0e (0x5f0f bytes) 000084[1] 1788510462.536157 trunc from 0x95598f to 0x0c54c5 (0x8904c9 bytes) 000085[0] 1788510462.578326 mapread 0x025000 thru 0x028a90 (0x3a91 bytes) 000086[0] 1788510462.617599 read 0x0a9000 thru 0x0b5ca8 (0xcca9 bytes) 000087[0] 1788510462.663589 mapread 0x040000 thru 0x048b1e (0x8b1f bytes) 000088[0] 1788510462.664936 mapwrite 0x129000 thru 0x1353f1 (0xc3f2 bytes) 000089[0] 1788510462.816807 punch from 0x042243 to 0x04b2fa (0x90b8 bytes) 000090[0] 1788510462.860959 read 0x0a6000 thru 0x0b5dca (0xfdcb bytes) truncating to largest ever: 0x97ffb0 000094[0] 1788510462.943180 trunc from 0x1353f2 to 0x97ffaf (0x84abbe bytes) 000095[0] 1788510462.999158 mapwrite 0x6db000 thru 0x6e3ed9 (0x8eda bytes) 000096[0] 1788510463.235168 mapread 0x379000 thru 0x382188 (0x9189 bytes) 000100[1] 1788510463.281364 trunc from 0x97ffb0 to 0x5ca080 (0x3b5f2f bytes) 000101[1] 1788510463.368676 mapread 0x0e7000 thru 0x0ef4dd (0x84de bytes) 000104[0] 1788510463.429467 mapread 0x212000 thru 0x214642 (0x2643 bytes) 000105[1] 1788510463.476617 write 0x44b000 thru 0x45a246 (0xf247 bytes) 000106[1] 1788510463.512665 mapread 0x344000 thru 0x34a2d2 (0x62d3 bytes) 000107[0] 1788510463.621657 mapread 0x085000 thru 0x094498 (0xf499 bytes) 000112[1] 1788510463.796142 mapwrite 0x93a000 thru 0x9477db (0xd7dc bytes) 000114[0] 1788510463.956026 read 0x87b000 thru 0x87e44e (0x344f bytes) 000116[0] 1788510463.984611 write 0x36f000 thru 0x36f476 (0x477 bytes) 000117[1] 1788510464.040382 trunc from 0x9477dc to 0x1cfbd0 (0x777c0b bytes) 000118[1] 1788510464.112965 write 0x9a8000 thru 0x9aaec5 (0x2ec6 bytes) 000119[0] 1788510464.159967 read 0x852000 thru 0x85a8dd (0x88de bytes) 000120[1] 1788510464.213370 punch from 0x26ad99 to 0x2759f7 (0xac5f bytes) 000122[1] 1788510464.281248 read 0x4f4000 thru 0x4fbe2a (0x7e2b bytes) 000124[0] 1788510464.360534 trunc from 0x9aaec6 to 0x92751f (0x839a6 bytes) 000125[0] 1788510464.415625 read 0x4a6000 thru 0x4a9e8e (0x3e8f bytes) 000126[0] 1788510464.448556 trunc from 0x927520 to 0x727add (0x1ffa42 bytes) 000127[0] 1788510464.509076 mapwrite 0x8e4000 thru 0x8ef702 (0xb703 bytes) 000130[1] 1788510464.701518 read 0x0de000 thru 0x0e1761 (0x3762 bytes) 000131[1] 1788510464.731653 trunc from 0x8ef703 to 0x36b8de (0x583e24 bytes) 000133[1] 1788510464.777285 mapwrite 0x652000 thru 0x660b65 (0xeb66 bytes) 000136[1] 1788510464.965815 mapwrite 0x241000 thru 0x247847 (0x6848 bytes) 000138[1] 1788510465.122136 punch from 0x112333 to 0x117b34 (0x5802 bytes) 000139[0] 1788510465.160032 trunc from 0x660b66 to 0x8ab877 (0x24ad12 bytes) 000140[1] 1788510465.217275 read 0x38c000 thru 0x396795 (0xa796 bytes) 000144[1] 1788510465.248141 write 0x1b9000 thru 0x1c8ee0 (0xfee1 bytes) 000145[1] 1788510465.268021 write 0x4c4000 thru 0x4d0a7b (0xca7c bytes) 000147[0] 1788510465.275007 mapwrite 0x1a7000 thru 0x1b4b2d (0xdb2e bytes) 000149[1] 1788510465.392266 punch from 0x557b05 to 0x55caf9 (0x4ff5 bytes) 000153[1] 1788510465.461378 write 0x5c5000 thru 0x5cc801 (0x7802 bytes) 000154[1] 1788510465.468956 write 0x0e3000 thru 0x0ed059 (0xa05a bytes) 000160[0] 1788510465.536179 write 0x51e000 thru 0x52a1f3 (0xc1f4 bytes) 000161[1] 1788510465.568926 mapread 0x87e000 thru 0x88b5ef (0xd5f0 bytes) 000162[1] 1788510465.674260 write 0x256000 thru 0x262386 (0xc387 bytes) 000164[0] 1788510465.723994 punch from 0x097c2d to 0x0a4159 (0xc52d bytes) 000165[0] 1788510465.759818 mapwrite 0x1b8000 thru 0x1bfc65 (0x7c66 bytes) 000167[0] 1788510465.828004 write 0x184000 thru 0x18a955 (0x6956 bytes) 000170[1] 1788510465.868438 mapread 0x3d7000 thru 0x3dc766 (0x5767 bytes) 000171[1] 1788510465.944674 write 0x852000 thru 0x857e4f (0x5e50 bytes) 000173[1] 1788510465.994586 mapwrite 0x55c000 thru 0x55f98a (0x398b bytes) 000175[0] 1788510466.064290 mapread 0x741000 thru 0x74b871 (0xa872 bytes) 000176[1] 1788510466.104881 punch from 0x61f065 to 0x625e57 (0x6df3 bytes) 000177[1] 1788510466.136372 trunc from 0x8ab878 to 0x1b430d (0x6f756a bytes) 000181[1] 1788510466.222493 read 0x06d000 thru 0x077816 (0xa817 bytes) 000183[1] 1788510466.293557 trunc from 0x1b430e to 0x20315b (0x4ee4e bytes) 000185[0] 1788510466.373363 write 0x343000 thru 0x3507de (0xd7df bytes) 000186[1] 1788510466.412012 trunc from 0x3507df to 0x0af4fe (0x2a12e0 bytes) 000187[1] 1788510466.524617 mapwrite 0x95b000 thru 0x96407c (0x907d bytes) 000193[1] 1788510466.824381 write 0x2e4000 thru 0x2f363a (0xf63b bytes) 000194[1] 1788510466.826995 mapread 0x822000 thru 0x831b18 (0xfb19 bytes) 000198[1] 1788510466.867490 mapread 0x159000 thru 0x165dde (0xcddf bytes) 000199[1] 1788510466.950477 mapread 0x5a2000 thru 0x5a2033 (0x34 bytes) 000200[0] 1788510466.976033 punch from 0x90e632 to 0x90fea8 (0x1877 bytes) 000202[1] 1788510467.086061 read 0x900000 thru 0x9069dc (0x69dd bytes) 000203[0] 1788510467.141302 punch from 0x20d705 to 0x2198cf (0xc1cb bytes) 000205[0] 1788510467.187574 mapread 0x2b0000 thru 0x2b3825 (0x3826 bytes) 000206[0] 1788510467.214988 punch from 0x944e43 to 0x947871 (0x2a2f bytes) 000209[1] 1788510467.237177 mapread 0x73c000 thru 0x749445 (0xd446 bytes) 000210[0] 1788510467.287512 mapwrite 0x101000 thru 0x10c2ec (0xb2ed bytes) 000212[1] 1788510467.439266 mapread 0x33b000 thru 0x3421e8 (0x71e9 bytes) 000214[0] 1788510467.493329 write 0x9bd000 thru 0x9bd2bc (0x2bd bytes) 000216[1] 1788510467.525712 trunc from 0x9bd2bd to 0x486909 (0x5369b3 bytes) 000217[1] 1788510467.562680 punch from 0x45f86f to 0x467ae6 (0x8278 bytes) 000218[1] 1788510467.568010 write 0x94e000 thru 0x956c70 (0x8c71 bytes) 000221[0] 1788510467.638571 write 0x806000 thru 0x813a95 (0xda96 bytes) 000222[0] 1788510467.677617 mapwrite 0x972000 thru 0x9780ba (0x60bb bytes) 000223[0] 1788510467.796791 mapwrite 0x67a000 thru 0x688359 (0xe35a bytes) 000224[0] 1788510467.950569 punch from 0x24d1ae to 0x25139f (0x41f2 bytes) truncating to largest ever: 0x9ea829 000226[1] 1788510467.985338 trunc from 0x9780bb to 0x9ea828 (0x7276e bytes) 000228[1] 1788510468.042676 mapread 0x20c000 thru 0x20d638 (0x1639 bytes) 000230[0] 1788510468.057572 mapread 0x778000 thru 0x77c871 (0x4872 bytes) 000232[0] 1788510468.091190 write 0x70e000 thru 0x71c70d (0xe70e bytes) 000233[0] 1788510468.115981 write 0x9b2000 thru 0x9bccae (0xacaf bytes) 000236[0] 1788510468.147777 mapwrite 0x991000 thru 0x9925a9 (0x15aa bytes) 000239[0] 1788510468.219052 punch from 0x51ba9a to 0x529624 (0xdb8b bytes) 000245[1] 1788510468.268787 trunc from 0x9ea829 to 0x6a45c1 (0x346267 bytes) 000246[1] 1788510468.323032 write 0x450000 thru 0x45c62d (0xc62e bytes) 000248[1] 1788510468.353410 trunc from 0x6a45c2 to 0x4d75b9 (0x1cd008 bytes) 000249[1] 1788510468.371099 write 0x758000 thru 0x763f8e (0xbf8f bytes) 000250[0] 1788510468.377305 punch from 0x6f2953 to 0x6f7607 (0x4cb5 bytes) 000251[1] 1788510468.459178 read 0x2e9000 thru 0x2ecc2a (0x3c2b bytes) 000253[0] 1788510468.499416 write 0x2de000 thru 0x2ed638 (0xf639 bytes) 000254[0] 1788510468.540281 punch from 0x4e7c5c to 0x4ec4b8 (0x485d bytes) 000255[0] 1788510468.560981 mapwrite 0x769000 thru 0x76d708 (0x4709 bytes) 000256[0] 1788510468.683042 mapwrite 0x791000 thru 0x79beb7 (0xaeb8 bytes) 000258[1] 1788510468.884198 punch from 0x1e3f83 to 0x1eca2f (0x8aad bytes) 000259[0] 1788510468.916452 trunc from 0x79beb8 to 0x6a85f6 (0xf38c1 bytes) 000260[0] 1788510468.964947 read 0x34f000 thru 0x3524c7 (0x34c8 bytes) 000261[1] 1788510468.989582 trunc from 0x6a85f7 to 0x7f57be (0x14d1c8 bytes) 000264[0] 1788510469.086766 write 0x53c000 thru 0x545e3a (0x9e3b bytes) 000265[1] 1788510469.110320 read 0x51c000 thru 0x526611 (0xa612 bytes) 000266[0] 1788510469.150210 read 0x44a000 thru 0x457063 (0xd064 bytes) 000267[0] 1788510469.168825 read 0x17a000 thru 0x17e555 (0x4556 bytes) 000268[0] 1788510469.192124 trunc from 0x7f57bf to 0x198a3f (0x65cd7f bytes) 000270[0] 1788510469.265644 write 0x160000 thru 0x16acac (0xacad bytes) 000273[1] 1788510469.310550 punch from 0x0c8bf8 to 0x0cd3d9 (0x47e2 bytes) 000274[1] 1788510469.339650 mapread 0x182000 thru 0x18cc32 (0xac33 bytes) 000276[1] 1788510469.398012 mapwrite 0x626000 thru 0x62fa05 (0x9a06 bytes) 000277[0] 1788510469.568891 mapwrite 0x1ee000 thru 0x1f588a (0x788b bytes) 000278[0] 1788510469.734954 write 0x1f3000 thru 0x20147f (0xe480 bytes) 000279[1] 1788510469.771813 read 0x40e000 thru 0x414fd2 (0x6fd3 bytes) 000280[1] 1788510469.807165 trunc from 0x62fa06 to 0x731d26 (0x102321 bytes) 000283[1] 1788510469.844409 mapwrite 0x600000 thru 0x60bb90 (0xbb91 bytes) 000284[1] 1788510469.952438 punch from 0x55c66f to 0x56377e (0x7110 bytes) 000286[0] 1788510470.010017 write 0x5b9000 thru 0x5bba19 (0x2a1a bytes) 000287[0] 1788510470.059107 mapread 0x21d000 thru 0x2292fb (0xc2fc bytes) 000288[0] 1788510470.108172 punch from 0x658339 to 0x65aee3 (0x2bab bytes) 000290[0] 1788510470.116123 punch from 0x5a071d to 0x5a4bdd (0x44c1 bytes) 000292[1] 1788510470.126752 read 0x160000 thru 0x165879 (0x587a bytes) 000293[1] 1788510470.152749 mapwrite 0x84f000 thru 0x84f919 (0x91a bytes) 000296[0] 1788510470.259960 punch from 0x357d73 to 0x367250 (0xf4de bytes) 000298[1] 1788510470.317751 mapwrite 0x307000 thru 0x30f941 (0x8942 bytes) 000302[0] 1788510470.557384 write 0x158000 thru 0x16048d (0x848e bytes) 000304[1] 1788510470.603207 mapread 0x2be000 thru 0x2c9aa7 (0xbaa8 bytes) 000305[1] 1788510470.718680 read 0x11b000 thru 0x121524 (0x6525 bytes) 000307[0] 1788510470.758381 trunc from 0x84f91a to 0x1b2732 (0x69d1e7 bytes) 000308[1] 1788510470.813089 mapwrite 0x49a000 thru 0x49be68 (0x1e69 bytes) 000310[0] 1788510470.911198 trunc from 0x49be69 to 0x62abe2 (0x18ed7a bytes) 000311[0] 1788510470.971391 mapread 0x440000 thru 0x440d54 (0xd55 bytes) 000312[1] 1788510471.011660 mapwrite 0x6c7000 thru 0x6ca8fb (0x38fc bytes) 000313[0] 1788510471.119356 write 0x1b0000 thru 0x1b9365 (0x9366 bytes) 000314[1] 1788510471.155489 punch from 0x508784 to 0x51333b (0xabb8 bytes) 000316[0] 1788510471.190181 punch from 0x10234e to 0x10f789 (0xd43c bytes) 000317[0] 1788510471.218767 punch from 0x512285 to 0x51b897 (0x9613 bytes) 000319[1] 1788510471.227115 trunc from 0x6ca8fc to 0x32caf8 (0x39de03 bytes) 000322[1] 1788510471.278550 write 0x20c000 thru 0x20c660 (0x661 bytes) 000325[0] 1788510471.328064 mapwrite 0x5a5000 thru 0x5ac5ea (0x75eb bytes) 000327[0] 1788510471.568203 mapwrite 0x04f000 thru 0x056f05 (0x7f06 bytes) 000328[0] 1788510471.686883 trunc from 0x5ac5eb to 0x4e0c93 (0xcb957 bytes) 000329[1] 1788510471.719138 mapread 0x070000 thru 0x078c70 (0x8c71 bytes) 000330[1] 1788510471.795223 mapwrite 0x27a000 thru 0x27a2b7 (0x2b8 bytes) 000331[1] 1788510471.917705 punch from 0x1f26da to 0x1fb0e1 (0x8a08 bytes) 000332[1] 1788510471.954150 punch from 0x464060 to 0x466fd7 (0x2f78 bytes) 000333[1] 1788510471.987161 read 0x483000 thru 0x4834a6 (0x4a7 bytes) 000334[0] 1788510472.003417 punch from 0x37af39 to 0x382d04 (0x7dcc bytes) 000336[1] 1788510472.041655 write 0x8bc000 thru 0x8bf395 (0x3396 bytes) 000337[0] 1788510472.077935 mapread 0x588000 thru 0x591210 (0x9211 bytes) 000338[1] 1788510472.133252 mapwrite 0x074000 thru 0x08032a (0xc32b bytes) 000339[0] 1788510472.227501 mapwrite 0x2ee000 thru 0x2f1b1b (0x3b1c bytes) 000340[0] 1788510472.339514 read 0x562000 thru 0x56de9c (0xbe9d bytes) 000342[0] 1788510472.383136 mapwrite 0x311000 thru 0x319f9d (0x8f9e bytes) 000346[1] 1788510472.489401 mapread 0x485000 thru 0x48975f (0x4760 bytes) 000349[0] 1788510472.530815 mapread 0x5bb000 thru 0x5c635a (0xb35b bytes) 000355[0] 1788510472.568379 read 0x53c000 thru 0x54633d (0xa33e bytes) 000357[0] 1788510472.580125 read 0x172000 thru 0x177d4f (0x5d50 bytes) 000358[1] 1788510472.592247 write 0x0a5000 thru 0x0b32c6 (0xe2c7 bytes) 000360[0] 1788510472.655045 mapread 0x503000 thru 0x50a375 (0x7376 bytes) 000362[1] 1788510472.740717 mapread 0x130000 thru 0x13a3c2 (0xa3c3 bytes) 000363[1] 1788510472.784973 read 0x370000 thru 0x3731c3 (0x31c4 bytes) 000364[0] 1788510472.794010 trunc from 0x8bf396 to 0x9a3ac0 (0xe472b bytes) 000367[0] 1788510472.876619 read 0x8d5000 thru 0x8e3dc2 (0xedc3 bytes) 000368[0] 1788510472.884757 read 0x61f000 thru 0x6221d5 (0x31d6 bytes) 000372[1] 1788510472.905492 mapread 0x5dd000 thru 0x5e9899 (0xc89a bytes) 000373[1] 1788510472.950361 mapwrite 0x909000 thru 0x90c38f (0x3390 bytes) 000374[0] 1788510473.033911 write 0x9fe000 thru 0x9ff5aa (0x15ab bytes) 000376[0] 1788510473.063593 write 0x8fd000 thru 0x903a5b (0x6a5c bytes) 000377[1] 1788510473.069393 write 0x4b0000 thru 0x4b0406 (0x407 bytes) 000378[1] 1788510473.096961 trunc from 0x9ff5ab to 0x95ac2f (0xa497b bytes) 000379[1] 1788510473.129219 write 0x0e0000 thru 0x0eec74 (0xec75 bytes) 000380[0] 1788510473.139655 trunc from 0x95ac30 to 0x9979f0 (0x3cdc1 bytes) 000381[1] 1788510473.200708 mapread 0x5a3000 thru 0x5a78a7 (0x48a8 bytes) 000383[0] 1788510473.252288 punch from 0x02954c to 0x0319a7 (0x845c bytes) 000384[0] 1788510473.279880 trunc from 0x9979f1 to 0x78b94b (0x20c0a5 bytes) 000385[0] 1788510473.317615 mapread 0x47a000 thru 0x486fef (0xcff0 bytes) 000388[1] 1788510473.424928 read 0x068000 thru 0x069b4a (0x1b4b bytes) 000390[0] 1788510473.454984 mapread 0x4ec000 thru 0x4f2202 (0x6203 bytes) 000391[1] 1788510473.467532 write 0x7f3000 thru 0x7ff152 (0xc153 bytes) 000392[0] 1788510473.487098 punch from 0x50da5b to 0x51051f (0x2ac5 bytes) 000393[1] 1788510473.577458 write 0x54b000 thru 0x555656 (0xa657 bytes) 000395[1] 1788510473.630647 mapwrite 0x2df000 thru 0x2e0d9a (0x1d9b bytes) 000396[0] 1788510473.727406 mapwrite 0x06f000 thru 0x070e44 (0x1e45 bytes) 000397[1] 1788510473.824416 write 0x7f9000 thru 0x806855 (0xd856 bytes) 000398[1] 1788510473.849677 punch from 0x6ca708 to 0x6ce019 (0x3912 bytes) 000400[0] 1788510473.853931 punch from 0x2b59e9 to 0x2c5912 (0xff2a bytes) 000401[1] 1788510473.889122 mapread 0x4e3000 thru 0x4ef98c (0xc98d bytes) 000404[1] 1788510473.937792 punch from 0x7c8d94 to 0x7cc289 (0x34f6 bytes) 000405[1] 1788510473.956776 mapread 0x06a000 thru 0x06d80c (0x380d bytes) 000406[0] 1788510473.987996 trunc from 0x806856 to 0x038ef2 (0x7cd963 bytes) 000407[0] 1788510474.030954 trunc from 0x038ef3 to 0x76f5cb (0x7366d9 bytes) 000409[0] 1788510474.070722 write 0x812000 thru 0x8217e9 (0xf7ea bytes) 000410[0] 1788510474.074584 mapread 0x5fb000 thru 0x60742b (0xc42c bytes) 000411[0] 1788510474.100317 read 0x6e7000 thru 0x6ed7a4 (0x67a5 bytes) 000412[1] 1788510474.127065 punch from 0x3843d1 to 0x390a87 (0xc6b7 bytes) 000414[0] 1788510474.167212 mapread 0x385000 thru 0x387f0b (0x2f0c bytes) 000415[1] 1788510474.208519 read 0x5ec000 thru 0x5ee165 (0x2166 bytes) 000418[1] 1788510474.220580 mapread 0x439000 thru 0x43d275 (0x4276 bytes) 000419[0] 1788510474.248400 write 0x64c000 thru 0x655031 (0x9032 bytes) 000420[1] 1788510474.287628 trunc from 0x8217ea to 0x1e17ce (0x64001b bytes) 000421[1] 1788510474.344391 read 0x1c3000 thru 0x1c57c6 (0x27c7 bytes) 000422[1] 1788510474.380039 mapread 0x181000 thru 0x18a69a (0x969b bytes) 000426[0] 1788510474.409101 trunc from 0x1e17cf to 0x395958 (0x1b418a bytes) 000428[1] 1788510474.470116 write 0x3f1000 thru 0x3fd2b7 (0xc2b8 bytes) 000429[1] 1788510474.487698 mapwrite 0x6d2000 thru 0x6d38ab (0x18ac bytes) 000430[0] 1788510474.697681 trunc from 0x6d38ac to 0x10b8b4 (0x5c7ff7 bytes) 000431[0] 1788510474.755703 read 0x086000 thru 0x088169 (0x216a bytes) 000432[1] 1788510474.788326 mapwrite 0x366000 thru 0x36c592 (0x6593 bytes) 000433[1] 1788510474.997296 write 0x40f000 thru 0x41c8ef (0xd8f0 bytes) 000434[1] 1788510475.020890 trunc from 0x41c8f0 to 0x4893cf (0x6cae0 bytes) 000435[1] 1788510475.064627 write 0x286000 thru 0x28a31b (0x431c bytes) 000437[0] 1788510475.111097 trunc from 0x4893d0 to 0x0c4a1e (0x3c49b1 bytes) 000439[0] 1788510475.184905 mapread 0x0b4000 thru 0x0c27ab (0xe7ac bytes) 000440[1] 1788510475.308358 read 0x081000 thru 0x0832fe (0x22ff bytes) 000441[0] 1788510475.382034 mapread 0x008000 thru 0x013264 (0xb265 bytes) 000444[0] 1788510475.439141 trunc from 0x0c4a1f to 0x9a2fbf (0x8de5a1 bytes) 000445[1] 1788510475.473459 punch from 0x584ea9 to 0x591c6d (0xcdc5 bytes) 000446[0] 1788510475.502437 read 0x1ae000 thru 0x1b5d69 (0x7d6a bytes) 000450[1] 1788510475.533539 read 0x513000 thru 0x51c6b0 (0x96b1 bytes) 000452[0] 1788510475.547099 punch from 0x44c60f to 0x45289f (0x6291 bytes) 000455[1] 1788510475.596928 punch from 0x3afc72 to 0x3b3fa7 (0x4336 bytes) 000456[0] 1788510475.633105 punch from 0x80ae6f to 0x80cde2 (0x1f74 bytes) 000457[1] 1788510475.667346 mapread 0x770000 thru 0x779e33 (0x9e34 bytes) 000458[1] 1788510475.733177 punch from 0x6f3506 to 0x701f87 (0xea82 bytes) 000459[1] 1788510475.751057 punch from 0x81323f to 0x817f72 (0x4d34 bytes) 000460[1] 1788510475.758556 trunc from 0x9a2fc0 to 0x9ae0c1 (0xb102 bytes) 000461[1] 1788510475.788501 mapread 0x35a000 thru 0x367944 (0xd945 bytes) 000462[1] 1788510475.836370 mapwrite 0x068000 thru 0x077b44 (0xfb45 bytes) 000463[1] 1788510475.978121 trunc from 0x9ae0c2 to 0x49bdad (0x512314 bytes) 000464[1] 1788510476.049277 write 0x868000 thru 0x8684da (0x4db bytes) 000466[0] 1788510476.067168 write 0x50b000 thru 0x5140f3 (0x90f4 bytes) 000467[0] 1788510476.111369 read 0x701000 thru 0x706847 (0x5848 bytes) 000468[1] 1788510476.118451 mapwrite 0x9cc000 thru 0x9d4836 (0x8837 bytes) 000469[1] 1788510476.224128 trunc from 0x9d4837 to 0x5f8a63 (0x3dbdd3 bytes) 000470[0] 1788510476.253268 mapread 0x38e000 thru 0x39c523 (0xe524 bytes) 000471[0] 1788510476.322928 write 0x0a1000 thru 0x0ad657 (0xc658 bytes) 000472[0] 1788510476.376505 write 0x17a000 thru 0x189ba6 (0xfba7 bytes) 000473[1] 1788510476.386667 punch from 0x51db6a to 0x528f2c (0xb3c3 bytes) 000475[0] 1788510476.431309 write 0x5d0000 thru 0x5de50c (0xe50d bytes) 000476[1] 1788510476.468634 mapwrite 0x716000 thru 0x722eef (0xcef0 bytes) 000477[0] 1788510476.616112 trunc from 0x722ef0 to 0x6addec (0x75103 bytes) 000478[1] 1788510476.648342 mapwrite 0x925000 thru 0x932ef5 (0xdef6 bytes) 000480[0] 1788510476.807237 trunc from 0x932ef6 to 0x510ccf (0x422226 bytes) 000481[0] 1788510476.845354 punch from 0x49d9c2 to 0x49e26f (0x8ae bytes) 000485[1] 1788510476.905206 mapwrite 0x720000 thru 0x726777 (0x6778 bytes) 000488[1] 1788510477.096472 trunc from 0x726778 to 0x3946c0 (0x3920b7 bytes) 000489[1] 1788510477.140416 read 0x0d7000 thru 0x0d8258 (0x1259 bytes) 000490[1] 1788510477.281063 punch from 0x1b08fd to 0x1b865d (0x7d61 bytes) 000491[1] 1788510477.335811 write 0x006000 thru 0x01448e (0xe48f bytes) 000492[1] 1788510477.338331 mapread 0x218000 thru 0x21fd06 (0x7d07 bytes) 000493[0] 1788510477.342685 write 0x868000 thru 0x874c24 (0xcc25 bytes) 000494[0] 1788510477.444543 write 0x290000 thru 0x29aa0c (0xaa0d bytes) 000495[0] 1788510477.499006 punch from 0x80016c to 0x80ae8d (0xad22 bytes) 000496[1] 1788510477.578070 trunc from 0x874c25 to 0x4851a8 (0x3efa7c bytes) 000497[0] 1788510477.683753 write 0x049000 thru 0x04c8bc (0x38bd bytes) All operations completed A-OK! PASS 16a (59s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 04:28:04 (1788510484) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.776368 s, 10.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.559781 s, 15.0 MB/s Chance of close/open is 1 in 50 Seed set to 489 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1654f6 000004[0] 1788510490.485547 trunc from 00000000 to 0x1654f5 (0x1654f6 bytes) 000005[1] 1788510490.589090 mapread 0x0f192f thru 0x0f29d4 (0x10a6 bytes) 000006[1] 1788510490.652561 punch from 0x122727 to 0x12323e (0xb18 bytes) truncating to largest ever: 0x1d9a7b 000007[1] 1788510490.717727 trunc from 0x1654f6 to 0x1d9a7a (0x74585 bytes) 000008[1] 1788510490.779416 mapread 0x1c659f thru 0x1c8423 (0x1e85 bytes) 000009[1] 1788510490.861376 punch from 0x16a22b to 0x16c13d (0x1f13 bytes) 000010[1] 1788510490.874744 punch from 0x189b94 to 0x18aad6 (0xf43 bytes) 000011[1] 1788510490.881499 read 0x109a6a thru 0x10a36f (0x906 bytes) 000012[0] 1788510490.888700 write 0x4d6834 thru 0x4d7e33 (0x1600 bytes) truncating to largest ever: 0x59774d 000014[0] 1788510490.958465 trunc from 0x4d7e34 to 0x59774c (0xbf919 bytes) 000015[0] 1788510491.020731 mapread 0x2eef1e thru 0x2f0948 (0x1a2b bytes) 000016[0] 1788510491.045373 mapread 0x0054fa thru 0x006c90 (0x1797 bytes) 000017[1] 1788510491.058768 punch from 0x19d3b2 to 0x19e1be (0xe0d bytes) 000018[0] 1788510491.123304 mapwrite 0x76503b thru 0x766865 (0x182b bytes) 000019[1] 1788510491.283122 punch from 0x1f08db to 0x1f1291 (0x9b7 bytes) 000020[0] 1788510491.320730 punch from 0x36120b to 0x36275d (0x1553 bytes) 000021[1] 1788510491.347585 mapread 0x6761c5 thru 0x676316 (0x152 bytes) 000022[0] 1788510491.402085 read 0x01c9d3 thru 0x01ce11 (0x43f bytes) 000025[1] 1788510491.460057 read 0x29add2 thru 0x29c180 (0x13af bytes) 000026[0] 1788510491.476056 trunc from 0x766866 to 0x4161d9 (0x35068c bytes) 000027[1] 1788510491.604290 mapread 0x2164e5 thru 0x216991 (0x4ad bytes) 000028[0] 1788510491.657885 read 0x1f6e09 thru 0x1f8048 (0x1240 bytes) 000029[1] 1788510491.675165 write 0x81be85 thru 0x81d4cb (0x1647 bytes) 000030[1] 1788510491.717825 mapwrite 0x69996d thru 0x69a1fa (0x88e bytes) 000033[1] 1788510491.826769 write 0x9dc3c5 thru 0x9dd488 (0x10c4 bytes) 000034[0] 1788510491.850895 read 0x382068 thru 0x38273d (0x6d6 bytes) 000035[0] 1788510491.898877 mapwrite 0x1407b7 thru 0x141d6e (0x15b8 bytes) 000037[1] 1788510492.023707 punch from 0x3e294f to 0x3e37d5 (0xe87 bytes) truncating to largest ever: 0x6bfd79 000038[0] 1788510492.061536 trunc from 0x9dd489 to 0x6bfd78 (0x31d710 bytes) 000039[1] 1788510492.108182 write 0x71c2b7 thru 0x71d4c2 (0x120c bytes) 000040[1] 1788510492.138048 read 0x0edad0 thru 0x0ee95c (0xe8d bytes) truncating to largest ever: 0x9e3dbd 000043[0] 1788510492.245011 trunc from 0x71d4c3 to 0x9e3dbc (0x2c68fa bytes) 000044[1] 1788510492.357604 write 0x07dfda thru 0x07ec40 (0xc67 bytes) 000045[1] 1788510492.409205 trunc from 0x9e3dbd to 0x9b04f3 (0x338c9 bytes) 000046[1] 1788510492.474732 trunc from 0x9b04f4 to 0x18d00c (0x8234e7 bytes) 000049[1] 1788510492.564268 trunc from 0x18d00d to 0x0a1a89 (0xeb583 bytes) 000051[1] 1788510492.623681 mapread 0x0469ea thru 0x047ece (0x14e5 bytes) 000052[0] 1788510492.690213 mapwrite 0x9e1152 thru 0x9e12af (0x15e bytes) 000053[1] 1788510492.822229 punch from 0x5c88b2 to 0x5c8947 (0x96 bytes) 000055[0] 1788510492.888215 punch from 0x29dc2d to 0x29f394 (0x1768 bytes) 000057[1] 1788510492.962080 mapread 0x8455f3 thru 0x845c3d (0x64b bytes) 000059[0] 1788510493.043727 mapwrite 0x7dde20 thru 0x7dedfc (0xfdd bytes) 000060[1] 1788510493.238901 write 0x948c0e thru 0x949060 (0x453 bytes) 000061[1] 1788510493.300676 mapwrite 0x9a5cde thru 0x9a69bf (0xce2 bytes) 000063[0] 1788510493.506317 write 0x8427b1 thru 0x843d34 (0x1584 bytes) 000067[0] 1788510493.659735 trunc from 0x9e12b0 to 0x5435c4 (0x49dceb bytes) 000068[1] 1788510493.748506 mapwrite 0x153e5a thru 0x15454a (0x6f1 bytes) 000070[1] 1788510493.947505 mapwrite 0x22f9f5 thru 0x22fe25 (0x431 bytes) 000074[0] 1788510494.111414 trunc from 0x5435c5 to 0x80dbda (0x2ca616 bytes) 000076[0] 1788510494.229970 punch from 0x12c852 to 0x12cc53 (0x402 bytes) 000077[0] 1788510494.255484 punch from 0x61a00c to 0x61bcb3 (0x1ca8 bytes) 000082[0] 1788510494.273971 trunc from 0x80dbdb to 0x93a7de (0x12cc04 bytes) 000083[1] 1788510494.314706 read 0x4c4f55 thru 0x4c50f9 (0x1a5 bytes) 000084[0] 1788510494.342112 write 0x1d8a22 thru 0x1d9ec4 (0x14a3 bytes) 000085[0] 1788510494.366883 mapread 0x55f704 thru 0x560147 (0xa44 bytes) 000086[0] 1788510494.390487 trunc from 0x93a7df to 0x385bba (0x5b4c24 bytes) 000090[0] 1788510494.490313 read 0x019c5c thru 0x01bbc6 (0x1f6b bytes) 000093[0] 1788510494.597078 mapwrite 0x104efa thru 0x1063c2 (0x14c9 bytes) 000094[0] 1788510494.663091 write 0x7a1631 thru 0x7a16bd (0x8d bytes) 000096[0] 1788510494.693946 write 0x319ac6 thru 0x31b9e2 (0x1f1d bytes) 000097[0] 1788510494.720134 mapread 0x1276d5 thru 0x1280fb (0xa27 bytes) 000099[0] 1788510494.776790 trunc from 0x7a16be to 0x1fdf83 (0x5a373a bytes) 000101[0] 1788510494.830950 mapread 0x12029c thru 0x12074b (0x4b0 bytes) 000102[0] 1788510494.849162 punch from 0x119316 to 0x11a9a7 (0x1692 bytes) 000103[0] 1788510494.880274 mapread 0x1e043d thru 0x1e145a (0x101e bytes) 000106[0] 1788510494.931908 read 0x03e7e4 thru 0x03ef73 (0x790 bytes) 000107[1] 1788510494.953452 read 0x1f5495 thru 0x1f6f46 (0x1ab2 bytes) 000110[1] 1788510495.025100 mapwrite 0x23f3e3 thru 0x24052d (0x114b bytes) 000111[1] 1788510495.150496 mapwrite 0x6a100b thru 0x6a2b4e (0x1b44 bytes) 000112[1] 1788510495.316930 trunc from 0x6a2b4f to 0x4fcad7 (0x1a6077 bytes) 000113[1] 1788510495.338375 mapwrite 0x858094 thru 0x858149 (0xb6 bytes) 000115[0] 1788510495.499217 write 0x3dd2de thru 0x3df226 (0x1f49 bytes) 000116[0] 1788510495.615500 punch from 0x2b675f to 0x2b858f (0x1e31 bytes) 000117[0] 1788510495.633355 mapwrite 0x887c4b thru 0x888c2a (0xfe0 bytes) 000119[0] 1788510495.828892 write 0x4d3cfa thru 0x4d46a1 (0x9a8 bytes) 000120[1] 1788510495.861159 read 0x4de327 thru 0x4de52b (0x205 bytes) 000122[1] 1788510495.916687 read 0x8316b3 thru 0x832c8a (0x15d8 bytes) 000123[0] 1788510495.948931 punch from 0x19093d to 0x1928ea (0x1fae bytes) 000125[0] 1788510495.970238 write 0x584eab thru 0x585ce1 (0xe37 bytes) 000126[1] 1788510496.010251 read 0x3143a9 thru 0x314aaf (0x707 bytes) 000127[0] 1788510496.071143 trunc from 0x888c2b to 0x88b4d7 (0x28ad bytes) 000129[1] 1788510496.218291 mapwrite 0x326db9 thru 0x3286d5 (0x191d bytes) 000130[1] 1788510496.541968 read 0x13f0f3 thru 0x140bcb (0x1ad9 bytes) 000134[0] 1788510496.723259 write 0x9824a9 thru 0x98373f (0x1297 bytes) 000135[0] 1788510496.754765 trunc from 0x983740 to 0x2c85ca (0x6bb175 bytes) 000136[1] 1788510496.848717 punch from 0x0fbcf6 to 0x0fd65a (0x1965 bytes) 000138[0] 1788510496.925104 mapread 0x07c4dd thru 0x07cd09 (0x82d bytes) 000139[1] 1788510496.978265 read 0x18e73e thru 0x18ea34 (0x2f7 bytes) 000142[1] 1788510497.188419 mapwrite 0x54b424 thru 0x54c7bf (0x139c bytes) 000146[0] 1788510497.617006 mapread 0x441a11 thru 0x441c75 (0x265 bytes) 000150[1] 1788510497.777185 write 0x3f28d8 thru 0x3f351d (0xc46 bytes) 000151[0] 1788510497.778179 mapwrite 0x5bfba6 thru 0x5c0b1d (0xf78 bytes) 000152[1] 1788510498.160541 mapwrite 0x4da808 thru 0x4db6e5 (0xede bytes) 000155[0] 1788510498.364956 mapwrite 0x65ca12 thru 0x65dfc8 (0x15b7 bytes) 000156[0] 1788510498.492466 punch from 0x07658f to 0x076dda (0x84c bytes) 000157[0] 1788510498.533090 mapwrite 0x5d28ad thru 0x5d4017 (0x176b bytes) 000158[0] 1788510498.781477 read 0x22cc36 thru 0x22d777 (0xb42 bytes) 000159[0] 1788510498.809392 mapwrite 0x7b3b24 thru 0x7b3dd1 (0x2ae bytes) 000160[1] 1788510499.006209 trunc from 0x7b3dd2 to 0x7a9220 (0xabb1 bytes) 000161[0] 1788510499.103401 punch from 0x534da7 to 0x53641f (0x1679 bytes) 000162[0] 1788510499.137375 mapread 0x38123c thru 0x3820f1 (0xeb6 bytes) 000163[1] 1788510499.193707 mapread 0x385ca3 thru 0x3864a4 (0x802 bytes) 000164[0] 1788510499.217205 mapread 0x470fd7 thru 0x472f2e (0x1f58 bytes) 000165[1] 1788510499.229133 punch from 0x08f475 to 0x0903a0 (0xf2c bytes) 000166[1] 1788510499.280320 punch from 0x69d6e8 to 0x69f2ad (0x1bc6 bytes) 000167[0] 1788510499.347139 mapwrite 0x3f306e thru 0x3f3495 (0x428 bytes) 000169[1] 1788510499.621235 trunc from 0x7a9221 to 0x567ba3 (0x24167d bytes) 000170[1] 1788510499.678973 trunc from 0x567ba4 to 0x741ce1 (0x1da13e bytes) 000172[0] 1788510499.756210 read 0x402176 thru 0x402f2e (0xdb9 bytes) 000173[0] 1788510499.797906 write 0x8abb86 thru 0x8ac7e5 (0xc60 bytes) 000176[0] 1788510499.908176 write 0x746a54 thru 0x747703 (0xcb0 bytes) 000180[1] 1788510500.003881 mapwrite 0x44fb54 thru 0x451553 (0x1a00 bytes) 000182[1] 1788510500.208639 trunc from 0x8ac7e6 to 0x7f03e7 (0xbc3fe bytes) 000183[0] 1788510500.382964 mapwrite 0x27df0c thru 0x27f0ec (0x11e1 bytes) 000184[1] 1788510500.633056 trunc from 0x7f03e8 to 0x7c6ff4 (0x293f3 bytes) 000185[1] 1788510500.707915 punch from 0x2d12d5 to 0x2d3299 (0x1fc5 bytes) 000186[1] 1788510500.716230 write 0x3b29ab thru 0x3b4095 (0x16eb bytes) 000187[0] 1788510500.733231 punch from 0x36e2fd to 0x36e3e2 (0xe6 bytes) 000189[1] 1788510500.879446 mapread 0x4f2135 thru 0x4f2c54 (0xb20 bytes) 000190[0] 1788510500.927468 trunc from 0x7c6ff5 to 0x67481c (0x1527d8 bytes) 000192[0] 1788510501.042201 mapwrite 0x5c4395 thru 0x5c51bb (0xe27 bytes) 000193[1] 1788510501.103971 punch from 0x3c79f2 to 0x3c9894 (0x1ea3 bytes) 000194[0] 1788510501.147688 trunc from 0x67481d to 0x965bd5 (0x2f13b9 bytes) 000198[0] 1788510501.270229 write 0x739892 thru 0x73b4c1 (0x1c30 bytes) 000202[1] 1788510501.348419 mapread 0x18c607 thru 0x18d8b8 (0x12b2 bytes) 000203[1] 1788510501.414022 punch from 0x050d85 to 0x0518c7 (0xb43 bytes) 000204[0] 1788510501.452773 mapread 0x8170e1 thru 0x8172ae (0x1ce bytes) 000205[1] 1788510501.495544 write 0x81cabd thru 0x81dd58 (0x129c bytes) 000207[0] 1788510501.598146 punch from 0x8bcef4 to 0x8bdd17 (0xe24 bytes) 000209[0] 1788510501.717582 trunc from 0x965bd6 to 0x4af65f (0x4b6576 bytes) 000210[0] 1788510501.785625 mapwrite 0x713ab7 thru 0x7152dd (0x1827 bytes) 000211[0] 1788510501.895673 punch from 0x5fb6a2 to 0x5fc860 (0x11bf bytes) 000212[0] 1788510501.921893 write 0x3de727 thru 0x3e016f (0x1a49 bytes) 000213[0] 1788510501.946029 write 0x497d0d thru 0x497f2d (0x221 bytes) 000214[1] 1788510501.987522 mapwrite 0x746896 thru 0x748773 (0x1ede bytes) 000215[1] 1788510502.239611 write 0x58aca4 thru 0x58c1b6 (0x1513 bytes) 000216[1] 1788510502.256221 mapwrite 0x5d5d84 thru 0x5d5dd0 (0x4d bytes) 000218[1] 1788510502.337032 mapwrite 0x3a12fe thru 0x3a1fc8 (0xccb bytes) 000220[0] 1788510502.461484 mapread 0x63c2ea thru 0x63deff (0x1c16 bytes) 000222[0] 1788510502.533014 mapread 0x6f1b20 thru 0x6f2a7b (0xf5c bytes) 000224[1] 1788510502.581762 trunc from 0x748774 to 0x081ada (0x6c6c99 bytes) 000225[0] 1788510502.622025 read 0x043159 thru 0x0432ca (0x172 bytes) 000226[0] 1788510502.658291 read 0x04cea4 thru 0x04d669 (0x7c6 bytes) 000227[1] 1788510502.706103 trunc from 0x081adb to 0x700abc (0x67efe2 bytes) 000228[1] 1788510502.757527 mapread 0x696a53 thru 0x698454 (0x1a02 bytes) 000229[0] 1788510502.774013 read 0x27bcbe thru 0x27c471 (0x7b4 bytes) 000233[0] 1788510502.900600 trunc from 0x700abd to 0x50843f (0x1f867d bytes) 000237[1] 1788510502.990735 mapread 0x3d4677 thru 0x3d634c (0x1cd6 bytes) 000238[0] 1788510503.030810 mapwrite 0x12d32c thru 0x12e64f (0x1324 bytes) 000242[1] 1788510503.182888 punch from 0x053344 to 0x054d29 (0x19e6 bytes) 000244[1] 1788510503.234109 trunc from 0x508440 to 0x6fbf28 (0x1f3ae9 bytes) 000245[0] 1788510503.269706 mapwrite 0x217220 thru 0x21896a (0x174b bytes) 000247[1] 1788510503.387697 write 0x45bde5 thru 0x45c963 (0xb7f bytes) 000250[1] 1788510503.459512 mapread 0x2aef75 thru 0x2af89f (0x92b bytes) 000251[1] 1788510503.501724 trunc from 0x6fbf29 to 0x6231f9 (0xd8d2f bytes) 000252[1] 1788510503.542800 read 0x45b3ad thru 0x45d1f9 (0x1e4d bytes) 000253[1] 1788510503.549119 read 0x27ff5b thru 0x281a83 (0x1b29 bytes) 000256[0] 1788510503.557475 punch from 0x1ef6be to 0x1f120a (0x1b4d bytes) 000257[1] 1788510503.589714 trunc from 0x6231fa to 0x4ba681 (0x168b78 bytes) 000258[0] 1788510503.656049 read 0x29cd19 thru 0x29e8a3 (0x1b8b bytes) 000259[0] 1788510503.685793 punch from 0x174fd4 to 0x1760d6 (0x1103 bytes) 000260[0] 1788510503.730985 write 0x28ac22 thru 0x28c7b2 (0x1b91 bytes) 000261[0] 1788510503.769164 read 0x3ebaa3 thru 0x3ebced (0x24b bytes) 000262[1] 1788510503.787903 mapwrite 0x506b5b thru 0x5078f5 (0xd9b bytes) 000263[0] 1788510503.886535 trunc from 0x5078f6 to 0x5fa291 (0xf299c bytes) 000266[0] 1788510503.990208 trunc from 0x5fa292 to 0x1c14f2 (0x438d9f bytes) 000267[1] 1788510504.021538 trunc from 0x1c14f3 to 0x686eec (0x4c59fa bytes) 000268[0] 1788510504.070179 read 0x373517 thru 0x373877 (0x361 bytes) 000269[1] 1788510504.198389 mapwrite 0x7d6993 thru 0x7d7f14 (0x1582 bytes) 000270[0] 1788510504.335525 punch from 0x0dd148 to 0x0def79 (0x1e32 bytes) 000272[0] 1788510504.397205 mapread 0x7ce4ac thru 0x7cfaeb (0x1640 bytes) 000273[0] 1788510504.594750 trunc from 0x7d7f15 to 0x49a930 (0x33d5e4 bytes) 000274[0] 1788510504.638476 trunc from 0x49a931 to 0x26449e (0x236492 bytes) 000283[1] 1788510504.812734 punch from 0x095cfa to 0x096255 (0x55c bytes) 000285[0] 1788510504.860634 mapwrite 0x280e04 thru 0x281841 (0xa3e bytes) 000287[1] 1788510505.002392 trunc from 0x281842 to 0x5ab9cb (0x32a18a bytes) 000288[1] 1788510505.038512 mapwrite 0x97aec7 thru 0x97bd1c (0xe56 bytes) 000289[0] 1788510505.151423 write 0x6d35a9 thru 0x6d4e32 (0x188a bytes) 000290[1] 1788510505.201208 trunc from 0x97bd1d to 0x93fccf (0x3c04d bytes) 000291[1] 1788510505.319905 punch from 0x0659a1 to 0x067841 (0x1ea1 bytes) 000292[1] 1788510505.366589 mapwrite 0x23faf6 thru 0x240d78 (0x1283 bytes) 000294[0] 1788510505.500513 trunc from 0x93fcd0 to 0x79ab1b (0x1a51b4 bytes) 000295[1] 1788510505.542773 mapread 0x2918d3 thru 0x292145 (0x873 bytes) 000298[0] 1788510505.614289 punch from 0x364b72 to 0x364e47 (0x2d6 bytes) 000299[0] 1788510505.651239 mapwrite 0x96ae4e thru 0x96c4cd (0x1680 bytes) 000300[0] 1788510505.937698 trunc from 0x96c4ce to 0x1f738f (0x77513e bytes) 000301[0] 1788510505.984115 mapwrite 0x92efd1 thru 0x92fb60 (0xb90 bytes) 000302[1] 1788510506.106171 read 0x457713 thru 0x458cb2 (0x15a0 bytes) 000303[1] 1788510506.132604 punch from 0x27b858 to 0x27c8b2 (0x105b bytes) 000306[0] 1788510506.167612 trunc from 0x92fb61 to 0x66bc09 (0x2c3f57 bytes) 000308[1] 1788510506.248370 punch from 0x4df03a to 0x4df658 (0x61f bytes) 000309[1] 1788510506.290359 read 0x21e184 thru 0x21fe71 (0x1cee bytes) 000310[1] 1788510506.383693 write 0x932e83 thru 0x93431a (0x1498 bytes) 000314[1] 1788510506.527185 read 0x08f644 thru 0x08fc8e (0x64b bytes) 000315[0] 1788510506.583726 mapread 0x5aa84b thru 0x5aaaa0 (0x256 bytes) 000316[0] 1788510506.667136 mapwrite 0x70ddec thru 0x70fbc2 (0x1dd7 bytes) 000317[1] 1788510506.882157 read 0x13c047 thru 0x13cf8f (0xf49 bytes) 000318[1] 1788510506.912467 punch from 0x32666f to 0x3268c2 (0x254 bytes) 000322[1] 1788510507.086330 mapwrite 0x572fd5 thru 0x573983 (0x9af bytes) 000324[0] 1788510507.291580 mapwrite 0x1a1bc4 thru 0x1a3059 (0x1496 bytes) 000325[1] 1788510507.617597 write 0x72e784 thru 0x72f1c0 (0xa3d bytes) 000326[0] 1788510507.693539 punch from 0x2ca4d6 to 0x2cb4d1 (0xffc bytes) 000331[0] 1788510507.866794 mapwrite 0x9d9d83 thru 0x9da402 (0x680 bytes) 000332[1] 1788510508.081864 write 0x46fb97 thru 0x46ff29 (0x393 bytes) 000333[1] 1788510508.147064 mapread 0x709217 thru 0x70ac1a (0x1a04 bytes) 000334[1] 1788510508.201170 trunc from 0x9da403 to 0x52874e (0x4b1cb4 bytes) 000336[0] 1788510508.323603 mapwrite 0x98fb48 thru 0x990b4c (0x1005 bytes) 000337[1] 1788510508.490311 mapwrite 0x66ed1f thru 0x66ee28 (0x10a bytes) 000338[1] 1788510508.698482 punch from 0x1fa16b to 0x1fa9c2 (0x858 bytes) 000339[1] 1788510508.721681 write 0x936351 thru 0x936fd5 (0xc85 bytes) 000341[0] 1788510508.779624 mapread 0x12dbfe thru 0x12de10 (0x213 bytes) 000345[0] 1788510508.918797 read 0x8c1261 thru 0x8c1555 (0x2f5 bytes) 000346[1] 1788510508.959092 mapwrite 0x52641e thru 0x52714f (0xd32 bytes) 000347[0] 1788510509.044376 mapwrite 0x570f73 thru 0x571e74 (0xf02 bytes) 000350[1] 1788510509.222050 read 0x3bf601 thru 0x3c1383 (0x1d83 bytes) 000351[0] 1788510509.236506 mapwrite 0x0f90bd thru 0x0f94f0 (0x434 bytes) 000352[0] 1788510509.364019 read 0x40d53c thru 0x40e329 (0xdee bytes) 000353[1] 1788510509.445045 read 0x91f069 thru 0x920488 (0x1420 bytes) 000354[0] 1788510509.494264 write 0x7eabb4 thru 0x7eaf04 (0x351 bytes) 000355[1] 1788510509.501670 mapwrite 0x738dba thru 0x7395cc (0x813 bytes) 000356[0] 1788510509.698371 read 0x889191 thru 0x88a97b (0x17eb bytes) 000357[1] 1788510509.714825 read 0x08c7ab thru 0x08e750 (0x1fa6 bytes) 000358[0] 1788510509.734064 mapwrite 0x469f3e thru 0x46bb44 (0x1c07 bytes) 000359[1] 1788510509.921689 trunc from 0x990b4d to 0x3544ec (0x63c660 bytes) 000361[0] 1788510510.018541 trunc from 0x3544ed to 0x1a1404 (0x1b30e8 bytes) 000364[1] 1788510510.074863 read 0x0b45a2 thru 0x0b5c36 (0x1695 bytes) 000365[0] 1788510510.193559 read 0x05e84d thru 0x05fe7e (0x1632 bytes) 000366[0] 1788510510.313697 trunc from 0x1a1405 to 0x277c68 (0xd6864 bytes) 000367[1] 1788510510.385958 trunc from 0x277c69 to 0x7672b8 (0x4ef650 bytes) 000369[1] 1788510510.482691 mapread 0x24d16a thru 0x24ef6c (0x1e03 bytes) 000370[0] 1788510510.563888 mapread 0x27c9d6 thru 0x27ceb4 (0x4df bytes) 000372[1] 1788510510.745777 punch from 0x3d2bc5 to 0x3d32af (0x6eb bytes) 000376[0] 1788510510.837027 mapread 0x0139a6 thru 0x014419 (0xa74 bytes) 000377[1] 1788510510.892803 trunc from 0x7672b9 to 0x22fc2b (0x53768d bytes) 000379[1] 1788510510.940457 read 0x1a39e2 thru 0x1a4463 (0xa82 bytes) 000380[1] 1788510511.034895 mapwrite 0x75e32f thru 0x75fee9 (0x1bbb bytes) 000381[1] 1788510511.269620 mapwrite 0x2cc8d4 thru 0x2cdd66 (0x1493 bytes) 000384[1] 1788510511.374667 read 0x45c783 thru 0x45e109 (0x1987 bytes) 000385[0] 1788510511.398363 punch from 0x501ab8 to 0x5028cf (0xe18 bytes) 000388[1] 1788510511.519648 punch from 0x04c42b to 0x04c87f (0x455 bytes) 000389[1] 1788510511.545539 write 0x8f3780 thru 0x8f4d2c (0x15ad bytes) 000390[1] 1788510511.562688 punch from 0x010974 to 0x01114c (0x7d9 bytes) 000392[0] 1788510511.606021 write 0x5842c8 thru 0x585757 (0x1490 bytes) 000393[1] 1788510511.738638 read 0x02d729 thru 0x02eb7e (0x1456 bytes) 000394[0] 1788510511.758503 trunc from 0x8f4d2d to 0x4ddc65 (0x4170c7 bytes) 000398[0] 1788510511.861448 punch from 0x27cc95 to 0x27e056 (0x13c2 bytes) 000400[0] 1788510511.914803 punch from 0x38afb7 to 0x38b628 (0x672 bytes) 000401[1] 1788510511.949163 punch from 0x095be0 to 0x09691b (0xd3c bytes) 000402[0] 1788510511.987121 mapread 0x0efcc8 thru 0x0f1a22 (0x1d5b bytes) 000404[0] 1788510512.029964 mapwrite 0x4ad231 thru 0x4ae12e (0xefe bytes) 000405[1] 1788510512.081382 write 0x66d7a1 thru 0x66f1a2 (0x1a02 bytes) 000408[0] 1788510512.099786 mapread 0x32224b thru 0x32400b (0x1dc1 bytes) 000409[0] 1788510512.136870 read 0x30f53b thru 0x30fb35 (0x5fb bytes) 000410[1] 1788510512.158749 trunc from 0x66f1a3 to 0x9b023c (0x34109a bytes) 000412[1] 1788510512.238593 write 0x3f2a01 thru 0x3f3042 (0x642 bytes) 000414[1] 1788510512.309218 read 0x56b905 thru 0x56d86c (0x1f68 bytes) 000417[1] 1788510512.370215 trunc from 0x9b023d to 0x0821bb (0x92e081 bytes) 000418[1] 1788510512.421460 read 0x06642e thru 0x066d57 (0x92a bytes) 000419[0] 1788510512.444810 punch from 0x013079 to 0x0132a0 (0x228 bytes) 000420[0] 1788510512.475197 write 0x8bb209 thru 0x8bc9f5 (0x17ed bytes) 000423[0] 1788510512.538250 mapwrite 0x36cb2d thru 0x36e5b6 (0x1a8a bytes) 000424[1] 1788510512.725137 read 0x4f9ebf thru 0x4fa1fb (0x33d bytes) 000425[1] 1788510512.788809 read 0x6f3f2a thru 0x6f51ff (0x12d6 bytes) 000426[0] 1788510512.866113 punch from 0x59b6fa to 0x59ba7e (0x385 bytes) 000428[1] 1788510512.938916 read 0x11822d thru 0x1187b7 (0x58b bytes) 000429[1] 1788510512.974723 write 0x5309d9 thru 0x53284b (0x1e73 bytes) 000430[0] 1788510513.007320 mapwrite 0x98b3c3 thru 0x98bdf2 (0xa30 bytes) 000432[0] 1788510513.283668 write 0x0d94e6 thru 0x0d98ea (0x405 bytes) 000433[1] 1788510513.324090 write 0x392693 thru 0x3938d2 (0x1240 bytes) 000435[1] 1788510513.430253 write 0x32601e thru 0x326320 (0x303 bytes) 000436[0] 1788510513.452116 mapread 0x0d09be thru 0x0d18a3 (0xee6 bytes) 000438[0] 1788510513.486095 mapwrite 0x46f472 thru 0x46faaf (0x63e bytes) 000439[1] 1788510513.534743 write 0x643f65 thru 0x644b41 (0xbdd bytes) 000443[0] 1788510513.626663 write 0x303cab thru 0x304a6b (0xdc1 bytes) 000444[0] 1788510513.686720 mapwrite 0x356d2d thru 0x357a68 (0xd3c bytes) 000447[0] 1788510513.878382 write 0x361e06 thru 0x362b9c (0xd97 bytes) 000448[0] 1788510513.924660 trunc from 0x98bdf3 to 0x571cf8 (0x41a0fa bytes) 000449[0] 1788510514.009253 write 0x1c84a4 thru 0x1c9907 (0x1464 bytes) 000451[0] 1788510514.082597 punch from 0x50e556 to 0x50e916 (0x3c1 bytes) 000452[0] 1788510514.094469 punch from 0x3b9d39 to 0x3bb94a (0x1c12 bytes) 000454[1] 1788510514.138419 punch from 0x12543d to 0x12562e (0x1f2 bytes) 000457[1] 1788510514.234511 trunc from 0x571cf9 to 0x70b0ef (0x1993f7 bytes) 000458[1] 1788510514.367148 mapread 0x038ab6 thru 0x039822 (0xd6d bytes) 000459[1] 1788510514.411566 mapread 0x5247bc thru 0x5248c7 (0x10c bytes) 000461[0] 1788510514.424411 read 0x2e3754 thru 0x2e555a (0x1e07 bytes) 000462[0] 1788510514.463073 trunc from 0x70b0f0 to 0x7d487f (0xc9790 bytes) 000464[1] 1788510514.561351 mapwrite 0x793291 thru 0x793415 (0x185 bytes) 000465[0] 1788510514.654183 write 0x1fb20e thru 0x1fba95 (0x888 bytes) 000467[1] 1788510514.711972 trunc from 0x7d4880 to 0x9777ef (0x1a2f70 bytes) 000469[0] 1788510514.889663 read 0x55447b thru 0x554999 (0x51f bytes) 000470[0] 1788510514.935629 trunc from 0x9777f0 to 0x27c6e9 (0x6fb106 bytes) 000471[1] 1788510515.041216 write 0x783942 thru 0x784daf (0x146e bytes) 000472[1] 1788510515.086430 punch from 0x5360d8 to 0x536cc0 (0xbe9 bytes) 000473[0] 1788510515.105392 read 0x190809 thru 0x192217 (0x1a0f bytes) 000475[1] 1788510515.150600 mapread 0x6de6ae thru 0x6df40f (0xd62 bytes) 000476[0] 1788510515.205537 mapwrite 0x09581e thru 0x09780b (0x1fee bytes) 000478[1] 1788510515.392663 mapread 0x1e7b4e thru 0x1e9a8e (0x1f41 bytes) 000480[1] 1788510515.542457 punch from 0x4aa12e to 0x4ab50d (0x13e0 bytes) 000483[1] 1788510515.600818 punch from 0x753869 to 0x75446d (0xc05 bytes) 000484[0] 1788510515.631167 read 0x539f54 thru 0x53af8c (0x1039 bytes) 000485[0] 1788510515.659048 write 0x86d37e thru 0x86efdf (0x1c62 bytes) 000487[0] 1788510515.755523 mapread 0x5c996f thru 0x5caf52 (0x15e4 bytes) 000489[0] 1788510515.842710 write 0x2191e1 thru 0x219af2 (0x912 bytes) 000491[0] 1788510515.881566 write 0x0006a6 thru 0x002348 (0x1ca3 bytes) 000492[1] 1788510515.897965 punch from 0x1b9a4d to 0x1bb393 (0x1947 bytes) 000493[0] 1788510515.930693 trunc from 0x86efe0 to 0x99a5e5 (0x12b606 bytes) 000494[0] 1788510516.005396 write 0x1122ba thru 0x1128fd (0x644 bytes) 000495[1] 1788510516.030075 write 0x83f6fa thru 0x8412be (0x1bc5 bytes) 000496[0] 1788510516.077213 mapread 0x0d37db thru 0x0d4f03 (0x1729 bytes) 000497[0] 1788510516.119216 read 0x779670 thru 0x779bd7 (0x568 bytes) 000498[1] 1788510516.144353 trunc from 0x99a5e6 to 0x1c9c23 (0x7d09c2 bytes) All operations completed A-OK! PASS 16b (37s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 04:28:42 (1788510522) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 04:28:47 (1788510527) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.496596 s, 16.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.410639 s, 20.4 MB/s ldlm.namespaces.MGC192.168.203.111@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8efe02fa7000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8efe124ca000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8efe124ca000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8efe124ca000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5866 s, 8.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.69394 s, 28.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.19394 s, 32.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.38726 s, 16.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.72502 s, 12.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.15141 s, 25.3 MB/s PASS 16d (53s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 04:29:40 (1788510580) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.403066 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180023 s, 5.8 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 04:29:49 (1788510589) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 04:30:17 (1788510617) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 04:30:45 (1788510645) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 04:30:54 (1788510654) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 04:31:01 (1788510661) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.51043 s, 17.4 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.67406 s, 15.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.55705 s, 7.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.07185 s, 12.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.64058 s, 9.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.17975 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.74042 s, 7.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.94791 s, 13.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.43475 s, 10.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.86068 s, 6.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.0573 s, 6.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.38923 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.54164 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.32367 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.50813 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.47611 s, 10.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.01446 s, 8.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.55675 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.84498 s, 6.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (43s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 04:31:44 (1788510704) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported : Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Sep 4 04:32:33 EDT 2026 + fsx 2 done: rc=0 Fri Sep 4 04:32:33 EDT 2026 + fsx 3 done: rc=0 Fri Sep 4 04:32:33 EDT 2026 + fsx 4 done: rc=0 Fri Sep 4 04:32:33 EDT 2026 + fsx 5 done: rc=0 Fri Sep 4 04:32:33 EDT 2026 PASS 16k (60s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 04:32:44 (1788510764) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 04:32:55 (1788510775) mmap test1: basic mmap operation (PASS, 0.17734s) mmap test2: MAP_PRIVATE not write back (PASS, 0.089602s) mmap test3: concurrent mmap ops on two nodes (PASS, 14.579s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5789s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.18025s) mmap test6: check mmap write/read content on two nodes (PASS, 0.25387s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.3085s) mmap test11: multi-thread shared mmap (PASS, 0.86958s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1074s) PASS 18 (32s) == sanityn test 19: test concurrent uncached read races ========================================================================= 04:33:27 (1788510807) oleg311-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 04:33:32 (1788510812) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 04:33:40 (1788510820) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (9s) == sanityn test 23: others should see updated atime while another read============================================================== 04:33:49 (1788510829) now is 1788510832 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 new atime is 1788510893 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 04:34:59 (1788510899) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 04:35:06 (1788510906) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 04:35:14 (1788510914) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 04:35:22 (1788510922) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 04:35:26 (1788510926) utime: good mknod times 1788510926 <= 1788510927 <= 1788510927 for /mnt/lustre/f26a utime: good mknod times 1788510926 <= 1788510927 <= 1788510927 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 04:35:33 (1788510933) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 04:35:43 (1788510943) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-04 04:35:45.000000000 -0400 Birth: 2026-09-04 04:35:46.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-04 04:35:45.000000000 -0400 Birth: 2026-09-04 04:35:46.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 04:35:53 (1788510953) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 04:36:02 (1788510962) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 04:36:09 (1788510969) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.111@tcp:/lustre 7542784 38912 7499776 1% /mnt/lustre2 PASS 31b (22s) == sanityn test 31r: open-rename(replace) race =========== 04:36:31 (1788510991) -rw-r--r-- 1 root root 0 Sep 4 04:36 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 04:36:40 (1788511000) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43156 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 04:36:49 (1788511009) PASS 31t (10s) == sanityn test 32b: lockless i/o ======================== 04:36:58 (1788511018) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 04:37:01 (1788511021) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.8374 s, 28.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.31769 s, 39.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.11596 s, 24.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.37409 s, 22.1 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.9126 s, 27.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.1365 s, 24.5 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37152 s, 38.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41178 s, 37.1 MB/s PASS 32c (31s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 04:37:33 (1788511053) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 04:37:37 (1788511057) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 04:37:41 (1788511061) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 04:37:44 (1788511064) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 04:37:47 (1788511067) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8efe02fa7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8efe02fa7000.ost_server_uuid in FULL state after 0 sec oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8efe02fa7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8efe02fa7000.ost_server_uuid in IDLE state after 0 sec oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8efe02fa7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8efe02fa7000.ost_server_uuid in IDLE state after 0 sec oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8efe02fa7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8efe02fa7000.ost_server_uuid in IDLE state after 0 sec oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8efe02fa7000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8efe02fa7000.ost_server_uuid in IDLE state after 0 sec oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8efe02fa7000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8efe02fa7000.ost_server_uuid in IDLE state after 0 sec PASS 34 (139s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 04:40:06 (1788511206) fail_loc=0x80000317 Race attempt 0 Wait for 50462 50497 for 60 sec... - open/close 676 (time 1788511223.10 total 10.02 last 67.47) - open/close 1368 (time 1788511233.11 total 20.03 last 69.15) - open/close 2103 (time 1788511243.11 total 30.04 last 73.44) - open/close 2792 (time 1788511253.13 total 40.05 last 68.81) - open/close 3473 (time 1788511263.13 total 50.05 last 68.06) total: 4000 open/close in 57.78 seconds: 69.23 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.111@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.203.111@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 04:41:17 (1788511277) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg311-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.03426 s, 25.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7445504) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.18159 s, 44.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7498752) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.28286 s, 40.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7489536) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.1636 s, 45.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.24499 s, 42.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7494656) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.94304 s, 27.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.40907 s, 37.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.18196 s, 44.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7494656) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.19075 s, 44.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43978 s, 36.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7491584) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.25175 s, 41.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7444480) after(7496704) PASS 36 (323s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 04:46:40 (1788511600) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 - create 1052 (time 1788511614.28 total 10.00 last 105.17) - create 2151 (time 1788511624.28 total 20.01 last 109.85) - create 3349 (time 1788511634.29 total 30.01 last 119.76) - create 4428 (time 1788511644.30 total 40.02 last 107.81) - create 5347 (time 1788511654.31 total 50.03 last 91.81) - create 6237 (time 1788511664.31 total 60.03 last 88.99) - create 7303 (time 1788511674.32 total 70.04 last 106.48) - create 8556 (time 1788511684.32 total 80.04 last 125.26) - create 9819 (time 1788511694.33 total 90.05 last 126.25) total: 10000 create in 91.45 seconds: 109.35 ops/second PASS 37 (102s) == sanityn test 39a: file mtime does not change after rename ========================================================== 04:48:23 (1788511703) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 04:48:30 (1788511710) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 04:48:37 (1788511717) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 04:48:45 (1788511725) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 04:48:53 (1788511733) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (20s) == sanityn test 40c: pdirops: link and others ======================================================================== 04:49:13 (1788511753) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (20s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 04:49:33 (1788511773) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (17s) == sanityn test 40e: pdirops: rename and others ======================================================================== 04:49:50 (1788511790) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 04:50:08 (1788511808) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 04:50:21 (1788511821) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 04:50:33 (1788511833) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (13s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 04:50:46 (1788511846) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 04:50:59 (1788511859) Conflict PASS 41e (12s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 04:51:11 (1788511871) Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 04:51:24 (1788511884) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 04:51:36 (1788511896) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 04:51:49 (1788511909) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1088s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 05:09:57 (1788512997) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (16s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 05:10:13 (1788513013) open: File exists Conflict PASS 42b (14s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 05:10:27 (1788513027) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 05:10:41 (1788513041) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 05:10:55 (1788513055) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (15s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 05:11:10 (1788513070) Conflict PASS 42f (15s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 05:11:24 (1788513084) Conflict PASS 42g (16s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 05:11:40 (1788513100) Conflict PASS 42h (16s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 05:11:56 (1788513116) PASS 43a (80s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 05:13:16 (1788513196) Conflict PASS 43b (12s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 05:13:28 (1788513208) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 05:13:42 (1788513222) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 05:13:56 (1788513236) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 05:14:10 (1788513250) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 05:14:25 (1788513265) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 05:14:38 (1788513278) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 05:14:51 (1788513291) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 05:14:55 (1788513295) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (139s) == sanityn test 43k: unlink vs create ==================== 05:17:14 (1788513434) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1245s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 05:37:59 (1788514679) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 4 05:38:08 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 05:38:13 (1788514693) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 05:38:27 (1788514707) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 05:38:42 (1788514722) Conflict PASS 44d (16s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 05:38:58 (1788514738) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 05:39:13 (1788514753) Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 05:39:29 (1788514769) Conflict PASS 44g (17s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 05:39:45 (1788514785) Conflict PASS 44h (18s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 05:40:04 (1788514804) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 05:40:08 (1788514808) PASS 45a (149s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 05:42:37 (1788514957) Conflict PASS 45b (14s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 05:42:51 (1788514971) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 05:43:05 (1788514985) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (16s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 05:43:21 (1788515001) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 05:43:35 (1788515015) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 05:43:49 (1788515029) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (16s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 05:44:05 (1788515045) Conflict PASS 45h (15s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 05:44:21 (1788515061) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 05:44:24 (1788515064) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1219s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 06:04:43 (1788516283) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (17s) == sanityn test 46b: pdirops: link vs create ======================================================================== 06:05:00 (1788516300) open: File exists Conflict PASS 46b (16s) == sanityn test 46c: pdirops: link vs link ======================================================================== 06:05:16 (1788516316) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (17s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 06:05:33 (1788516333) Conflict PASS 46d (15s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 06:05:48 (1788516348) Conflict PASS 46e (15s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 06:06:03 (1788516363) Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 06:06:18 (1788516378) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 06:06:33 (1788516393) Conflict PASS 46h (15s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 06:06:48 (1788516408) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 06:06:52 (1788516412) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 06:06:55 (1788516415) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 06:06:59 (1788516419) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (5s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 06:07:04 (1788516424) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 06:07:07 (1788516427) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 06:07:11 (1788516431) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 06:07:15 (1788516435) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 06:07:20 (1788516440) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.261982 s, 39.1 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (15s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 06:07:34 (1788516454) 0+1 records in 0+1 records out 158 bytes copied, 0.0450379 s, 3.5 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218825) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 06:07:43 (1788516463) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0323232 s, 31.7 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105003 s, 10.0 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 06:08:03 (1788516483) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0286264 s, 35.8 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 06:08:15 (1788516495) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867753 s, 12.1 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 06:08:23 (1788516503) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 06:08:33 (1788516513) llite.lustre-ffff8efe02fa7000.xattr_cache=1 llite.lustre-ffff8efe124ca000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 06:08:35.000000000 -0400 Modify: 2026-09-04 06:08:35.000000000 -0400 Change: 2026-09-04 06:08:35.000000000 -0400 Birth: 2026-09-04 06:08:36.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 06:08:41.000000000 -0400 Modify: 2026-09-04 06:08:42.000000000 -0400 Change: 2026-09-04 06:08:42.000000000 -0400 Birth: 2026-09-04 06:08:42.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 06:08:47.000000000 -0400 Modify: 2026-09-04 06:08:47.000000000 -0400 Change: 2026-09-04 06:08:47.000000000 -0400 Birth: 2026-09-04 06:08:49.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 06:08:47.000000000 -0400 Modify: 2026-09-04 06:08:48.000000000 -0400 Change: 2026-09-04 06:08:48.000000000 -0400 Birth: 2026-09-04 06:08:49.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 06:08:54.000000000 -0400 Modify: 2026-09-04 06:08:55.000000000 -0400 Change: 2026-09-04 06:08:55.000000000 -0400 Birth: 2026-09-04 06:08:55.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 06:08:54.000000000 -0400 Modify: 2026-09-04 06:08:55.000000000 -0400 Change: 2026-09-04 06:08:55.000000000 -0400 Birth: 2026-09-04 06:08:55.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 06:09:07 (1788516547) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (16s) == sanityn test 55b: rename vs unlink source dir ========= 06:09:23 (1788516563) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (15s) == sanityn test 55c: rename vs unlink orphan target dir == 06:09:38 (1788516578) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 06:09:57 (1788516597) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 06:10:12 (1788516612) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 06:10:15 (1788516615) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 06:10:37 (1788516637) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (24s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 06:11:01 (1788516661) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788516663 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.50032 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (133s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 06:13:15 (1788516795) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788516800 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.904253 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (149s) == sanityn test 60: Verify data_version behaviour ======== 06:15:45 (1788516945) PASS 60 (11s) == sanityn test 70a: cd directory && rm directory ======== 06:15:57 (1788516957) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 06:16:06 (1788516966) PASS 70b (8s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 06:16:14 (1788516974) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.030269 s, 2.2 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 06:16:18 (1788516978) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0142392 s, 4.6 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 06:16:21 (1788516981) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 06:16:24 (1788516984) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0193182 s, 3.4 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 06:16:28 (1788516988) llite.lustre-ffff8efe02fa7000.xattr_cache=1 llite.lustre-ffff8efe124ca000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (9s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 06:16:38 (1788516998) llite.lustre-ffff8efe02fa7000.xattr_cache=1 llite.lustre-ffff8efe124ca000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (8s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 06:16:45 (1788517005) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0107437 s, 95.3 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0250251 s, 40.9 kB/s 231656: get lock2 231657: get lock1 231656: done 231656: get lock1 231657: done 231657: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231657: cannot get lock2: Resource deadlock avoided 231657: exit rc=0 231656: done 231656: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 06:16:56 (1788517016) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0795413 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445292 s, 23.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232238 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (14s) == sanityn test 76: Verify MDT open_files listing ======== 06:17:10 (1788517030) opening files: ........ closing files: ........ PASS 76 (79s) == sanityn test 77a: check FIFO NRS policy =============== 06:18:29 (1788517109) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.402279 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106991 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128723 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150903 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582433 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171351 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194653 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104667 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230545 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263808 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38952 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731502 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10782 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35862 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422783 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161291 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172987 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453526 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446367 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443242 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892248 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0084507 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149825 s, 70.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127696 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00654075 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235109 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238725 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249567 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220892 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00999074 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050578 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146889 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472247 s, 2.2 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (10s) == sanityn test 77b: check CRR-N NRS policy ============== 06:18:39 (1788517119) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.503784 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052885 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771177 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889012 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185904 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164989 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147721 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259498 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106292 s, 9.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352306 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340241 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291499 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341222 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302926 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34505 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396932 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393617 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164548 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171385 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176732 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173775 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137306 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639532 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24782 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196506 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263657 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20942 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927025 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17466 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278452 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236143 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254142 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264505 s, 4.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=3 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.777981 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075096 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713041 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121773 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178174 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273829 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243543 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348186 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26681 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366047 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339487 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308313 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380947 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413266 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379705 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378294 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473323 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411297 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12858 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533121 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462664 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136637 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606367 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645135 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740312 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785731 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761807 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105236 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864327 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955732 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129579 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103404 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12835 s, 8.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (18s) == sanityn test 77c: check ORR NRS policy ================ 06:18:57 (1788517137) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.735009 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833856 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108081 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101412 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122509 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123796 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115566 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180119 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739565 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769699 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199545 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320619 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181662 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267858 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267354 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267644 s, 3.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433836 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100075 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370913 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162875 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19671 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00784618 s, 134 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152651 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152227 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088585 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805462 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141226 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210798 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125345 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304605 s, 34.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358946 s, 29.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424911 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303935 s, 34.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.986677 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772382 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979188 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271981 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237539 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348229 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403662 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414472 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420026 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37575 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448544 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441909 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491802 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.5121 s, 2.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.516557 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.541049 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983847 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690945 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756759 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20705 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452745 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150722 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00652393 s, 161 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151216 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493761 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.61107 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714636 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698773 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103088 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125844 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107688 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00242434 s, 433 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0076853 s, 136 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.403098 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912812 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939161 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431064 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123746 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149199 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184637 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100702 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193866 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23273 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302722 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283731 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219575 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275604 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363678 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374482 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405779 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00703452 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309331 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071213 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322292 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343004 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508018 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436742 s, 24.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657522 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00606575 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487062 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477528 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471634 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481534 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100588 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078478 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687552 s, 15.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (19s) == sanityn test 77d: check TRR nrs policy ================ 06:19:16 (1788517156) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.40515 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302149 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038961 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136962 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155565 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155102 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172249 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16109 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244837 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228693 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221734 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248681 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279573 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275669 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251983 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275354 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360842 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350245 s, 299 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00529344 s, 198 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142866 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936312 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063141 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898051 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182455 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704758 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692258 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953561 s, 11.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955275 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282284 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277042 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132898 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544254 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103802 s, 10.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.465105 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509803 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582485 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425121 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271076 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312849 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290552 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341197 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426329 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441024 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431204 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401021 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43835 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446217 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476918 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00940875 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.505308 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491888 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411038 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364261 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278805 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36203 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158093 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441292 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188083 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117164 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811049 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619551 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473542 s, 22.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542011 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441546 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231278 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302516 s, 34.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.770284 s, 21.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138153 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801419 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133489 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308073 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289513 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259732 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316323 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317583 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38069 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317099 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488211 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394112 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431121 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471355 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389809 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.507952 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076344 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371279 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932455 s, 11.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680772 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719642 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704198 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744758 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14086 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199052 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123604 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085533 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104219 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107973 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115932 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115922 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123903 s, 8.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 06:19:34 (1788517174) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.11@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.590366 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115608 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106579 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270315 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315646 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323256 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295752 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325585 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357164 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395085 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331144 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36297 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406938 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372553 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475998 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460753 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481909 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119893 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798817 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115072 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856815 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106136 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12455 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168523 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102463 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11735 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137192 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104499 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105362 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10937 s, 9.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115418 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10434 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115529 s, 9.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.359422 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657365 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108624 s, 9.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156406 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116642 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165954 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148029 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147309 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156464 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148463 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216556 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258711 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295074 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232612 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269479 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241714 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294128 s, 3.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255642 s, 410 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629884 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944624 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756197 s, 13.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405004 s, 259 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853223 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899678 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752643 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130579 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831351 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106226 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981842 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777547 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111519 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342044 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349837 s, 30.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.464962 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820181 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581284 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102862 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116384 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23403 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311165 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359916 s, 2.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387729 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334524 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332991 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397846 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387327 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402528 s, 2.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376771 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396652 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523683 s, 20.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212163 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155835 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19895 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169344 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725726 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173122 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45822 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117956 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552545 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333237 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394587 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975419 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13113 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100592 s, 10.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00311131 s, 337 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263903 s, 39.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.388621 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050147 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118146 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061464 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141599 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12908 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167221 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191354 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240567 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21688 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184884 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725098 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25505 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130134 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181934 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135751 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180855 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233312 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592669 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315595 s, 33.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605359 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380579 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436167 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487129 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121834 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531681 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931505 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576078 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805541 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674199 s, 15.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638575 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634903 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653094 s, 16.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (22s) == sanityn test 77f: check TBF JobID nrs policy ========== 06:19:56 (1788517196) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.598002 s, 28.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837829 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113868 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105483 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124068 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283504 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257552 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288053 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264773 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309042 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327668 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243996 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339192 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297318 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445822 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387387 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394067 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209464 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860588 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883688 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996487 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382692 s, 27.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497346 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100387 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587983 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568639 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699914 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767487 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775533 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784832 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907064 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873334 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108665 s, 9.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.653418 s, 25.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848565 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129077 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108918 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170364 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530465 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134323 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903805 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183277 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419675 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492121 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24926 s, 4.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171683 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259462 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213138 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30699 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.522461 s, 2.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100013 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150603 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189612 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120236 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013093 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182657 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210142 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215465 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171327 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140435 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197392 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674936 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516407 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569953 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365901 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013868 s, 75.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.635685 s, 26.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436782 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642089 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126412 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13173 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192364 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237133 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18275 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291351 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323032 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279505 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294346 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266738 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333117 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418757 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523462 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.605762 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294591 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255881 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148211 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167302 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131169 s, 8.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128604 s, 81.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21815 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113588 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969015 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945302 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128513 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088292 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562324 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387852 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761393 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924566 s, 11.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.430999 s, 38.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801509 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152486 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153641 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174809 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204258 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147839 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122501 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270033 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24057 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268539 s, 3.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388035 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.48512 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43922 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466588 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.527795 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.628762 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729775 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842903 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892438 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172978 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662818 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125838 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846935 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115249 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138272 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116785 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15189 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168691 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12362 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123109 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11201 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127001 s, 8.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (27s) == sanityn test 77g: Change TBF type directly ============ 06:20:23 (1788517223) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 06:20:34 (1788517234) oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg311-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg311-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg311-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg311-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 PASS 77h (10s) == sanityn test 77i: Change rank of TBF rule ============= 06:20:45 (1788517245) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (22s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 06:21:06 (1788517266) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.794651 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589092 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679288 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112773 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12769 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10725 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299127 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300346 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351622 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262748 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337563 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362193 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370687 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386998 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368213 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397647 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404062 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23189 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255401 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227816 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27484 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27549 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260415 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242274 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306531 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253205 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267112 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246182 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270053 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279055 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2809 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318138 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275469 s, 3.8 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.74206 s, 12.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6143 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (64s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 06:22:10 (1788517330) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg311-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg311-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg311-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg311-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29764 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649615 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138024 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10805 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19228 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195835 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204022 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223844 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153759 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229103 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117472 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257637 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241133 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222448 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258459 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306142 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056121 s, 18.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689131 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751404 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705861 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770376 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817174 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078739 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780499 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859265 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858653 s, 12.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572738 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939367 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831547 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104171 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074049 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137836 s, 7.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9413 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9529 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg311-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg311-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg311-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg311-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33738 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132049 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186353 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168621 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215156 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189979 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279106 s, 3.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154789 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318897 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12462 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223386 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327294 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225948 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390496 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322741 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289384 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351139 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816484 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975961 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847757 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819923 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103794 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704249 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729742 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629832 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101735 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941362 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099271 s, 10.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557727 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785113 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062383 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714374 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731191 s, 14.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2203 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0041 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (148s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 06:24:38 (1788517478) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg311-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg311-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg311-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg311-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32039 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088664 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102882 s, 10.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888403 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175281 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188318 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120679 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182835 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189383 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194017 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210808 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138305 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161688 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238758 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311625 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269703 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262232 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065857 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780668 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730166 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101505 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108984 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865752 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111808 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114658 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812307 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886594 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079468 s, 13.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114933 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10416 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132909 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13735 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157557 s, 6.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0038 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8391 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg311-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg311-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg311-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg311-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28877 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111871 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524894 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188857 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163523 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160863 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104313 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176351 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244334 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282837 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253172 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262802 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232568 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25397 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657757 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243696 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369639 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04559 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587131 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536724 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566268 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669862 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602773 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658224 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722633 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074649 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892361 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900332 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100538 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740661 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804083 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975491 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972578 s, 10.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9994 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9944 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (155s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 06:27:13 (1788517633) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.561598 s, 29.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432738 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437695 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512684 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799226 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890807 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19381 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247894 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2533 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256486 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247664 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288071 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294847 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255315 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290839 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298843 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314952 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027526 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599599 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420854 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061061 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678794 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731196 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727463 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707964 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366829 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00457421 s, 229 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796565 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067201 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456674 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868085 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638149 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637913 s, 16.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.9661 s, 7.0 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0498 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.11@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.11@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.793516 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413209 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749096 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548311 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982174 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167727 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27877 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311998 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194268 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28791 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317393 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302173 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252668 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217438 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226469 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351728 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319175 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937889 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413098 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105072 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443241 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108901 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128864 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571053 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577359 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898985 s, 11.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819005 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105587 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844938 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863777 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884642 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11414 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100332 s, 10.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2859 s, 10.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98198 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.11@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.521492 s, 32.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905632 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647281 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924107 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861643 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936561 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283317 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297479 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331579 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330025 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362126 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365506 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355722 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403174 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415411 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444999 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471562 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582514 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566868 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668994 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558266 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339105 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674657 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572652 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576234 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728456 s, 14.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509449 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587774 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690413 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708851 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647646 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554399 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731798 s, 14.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.46707 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12328 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.701551 s, 23.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060343 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719159 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939556 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885814 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710395 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110847 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663964 s, 15.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627748 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105008 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263701 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229531 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23992 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199636 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314838 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32139 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3387 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771407 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793066 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154509 s, 67.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172081 s, 60.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280318 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183893 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819944 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027069 s, 38.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335668 s, 31.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350668 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471129 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454023 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142056 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541798 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567933 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626936 s, 16.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.529 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0098 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6331 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95922 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.06121 s, 15.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923407 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101702 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517006 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999756 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162452 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114856 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104794 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236514 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245049 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181625 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113326 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120303 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326173 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351715 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254789 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231256 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15774 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162561 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08683 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132753 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15903 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154483 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191012 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107076 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984194 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878045 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11962 s, 8.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109013 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120755 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123455 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13637 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214164 s, 4.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9067 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6622 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.797852 s, 21.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116507 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125695 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136146 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117159 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220631 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208035 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246253 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236778 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179833 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252371 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152849 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3087 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20396 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230609 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900764 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296452 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787812 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145226 s, 72.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111902 s, 93.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335617 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0067036 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128297 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940186 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537732 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553681 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031823 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660818 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00662776 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403947 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278975 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281838 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303579 s, 34.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1195 s, 10.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95355 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.887074 s, 18.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030139 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059909 s, 17.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301084 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882588 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201733 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204255 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187724 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234406 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200248 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218481 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282393 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233327 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261759 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27491 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265344 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423268 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940312 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508773 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483862 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497071 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602863 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608047 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116235 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119559 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126874 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764058 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132775 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825733 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839044 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149758 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930663 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112879 s, 9.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9204 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99901 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.4046 s, 8.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94325 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (465s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 06:34:58 (1788518098) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.575715 s, 29.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837578 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103065 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269854 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24349 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247471 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313086 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296772 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302324 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291614 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29048 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349269 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340832 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37643 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355633 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391159 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465237 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780085 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740096 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331997 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439373 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584023 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548966 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512066 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624166 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583228 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662587 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690822 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707378 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584915 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769692 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680802 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732969 s, 14.3 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.62673 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92597 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (53s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 06:35:51 (1788518151) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.11@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.11@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.709329 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157939 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16758 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15398 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271228 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273451 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243069 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224086 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284694 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356698 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34025 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359263 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404318 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311202 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353743 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350117 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424809 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152588 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141413 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148963 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168819 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174965 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176453 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212606 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234661 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182982 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188074 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180116 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18518 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196283 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181787 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194168 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212065 s, 4.9 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95991 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0355 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (56s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 06:36:47 (1788518207) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.11@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.693969 s, 24.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899614 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176612 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242126 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257856 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211578 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302017 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295357 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309818 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366465 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388454 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40956 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352146 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387258 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420184 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350488 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.547267 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998245 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297557 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698377 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317443 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358685 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443762 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119316 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866293 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823873 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444953 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698883 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681042 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133921 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747037 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755217 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819192 s, 12.8 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.36871 s, 11.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23383 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (47s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 06:37:34 (1788518254) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.715868 s, 23.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607294 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104258 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967217 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691665 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162462 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134563 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201988 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231664 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165009 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252134 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223317 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274735 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246659 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259539 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258214 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299905 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256827 s, 408 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157026 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194333 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130298 s, 80.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142125 s, 73.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699155 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11769 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00718457 s, 146 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995078 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122946 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992141 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840059 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113681 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963216 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607948 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594355 s, 17.6 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8505 s, 8.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96038 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.2257 s, 8.6 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9839 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (101s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 06:39:15 (1788518355) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.09978 s, 15.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943678 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751028 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11299 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170514 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158439 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14262 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23891 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196636 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744464 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201854 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274587 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282859 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222806 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34027 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30496 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3141 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350283 s, 29.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150626 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583549 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557529 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510471 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759223 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631713 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710176 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00478622 s, 219 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138781 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058428 s, 179 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794188 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736085 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236612 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149028 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00888834 s, 118 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0052 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0307 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (80s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 06:40:35 (1788518435) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.983154 s, 17.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128621 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224655 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223102 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334353 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237671 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173945 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280683 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312847 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46711 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34959 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe02fa7000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.546981 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.572434 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.665443 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419096 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336178 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436029 s, 2.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe02fa7000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695212 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409816 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471285 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016605 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158269 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240509 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401204 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00351725 s, 298 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143747 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100483 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453217 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103835 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140354 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129711 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104755 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201345 s, 52.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.678 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0304 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg311-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.521248 s, 32.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905069 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815237 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991788 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207436 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995906 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14308 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216623 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226041 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208944 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201595 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220845 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269392 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297747 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109385 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15585 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213205 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799729 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884866 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11984 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897055 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920916 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983063 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954946 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736299 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716736 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115963 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770614 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929104 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973345 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977085 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14348 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130054 s, 8.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4609 s, 9.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0636 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1269 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94357 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (145s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 06:43:00 (1788518580) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg311-client.virtnet /mnt/lustre (opts:) Stopping client oleg311-client.virtnet /mnt/lustre opts: Stopping clients: oleg311-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg311-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg311-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg311-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.11,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg311-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.11,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg311-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.11,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69480/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.11,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg311-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: kmemleak disabled oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: detected 4 online CPUs by sysfs oleg311-server: MODOPTS_LIBCFS= oleg311-server: Force libcfs to create 2 CPU partitions oleg311-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre Starting client oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre Started clients oleg311-client.virtnet: 192.168.203.111@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Starting client oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Started clients oleg311-client.virtnet: 192.168.203.111@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8efe03500800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8efe1ecef000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8efe03500800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8efe1ecef000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.450532 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415843 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104119 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110536 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782561 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866827 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121329 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088089 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117247 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212104 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224741 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112939 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221517 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147416 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159172 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304397 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30633 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00238346 s, 440 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826257 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108885 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00765447 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00483526 s, 217 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291807 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00856084 s, 122 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813517 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106449 s, 98.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870576 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00609801 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131847 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269161 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012053 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108093 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025064 s, 41.8 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.31159 s, 11.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17666 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.569619 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122971 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136316 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270925 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298411 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292598 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345482 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331123 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394241 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328869 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392957 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367669 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341187 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440094 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451181 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354485 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387971 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936664 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904711 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969259 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891615 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061933 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117927 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138575 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663593 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119185 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106653 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798789 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892452 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136762 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920059 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15619 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109382 s, 9.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0973 s, 10.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0968 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (223s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 06:46:43 (1788518803) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg311-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 06:47:01 (1788518821) On MGS 192.168.203.111, active = nodemap.active=1 On MGS 192.168.203.111, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.865471 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567325 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801361 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765013 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484429 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863943 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24803 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181051 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151677 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251128 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249084 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225429 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136404 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200067 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659056 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191154 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306876 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367545 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104074 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189925 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181196 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115036 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121969 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319991 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576034 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035997 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127312 s, 82.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136714 s, 76.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115273 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271893 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131636 s, 79.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241057 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441031 s, 23.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1108 s, 10.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.31372 s, 19.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.111, TBF.id = On MGS 192.168.203.111, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.706928 s, 23.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107726 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126381 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107534 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180029 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169651 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198135 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199886 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189983 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163439 s, 6.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe1ecef000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27941 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269389 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364409 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36296 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367126 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448227 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.497569 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16975 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173016 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177736 s, 5.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176674 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189265 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195076 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194259 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206276 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156542 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158957 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14432 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119969 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167801 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171019 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243868 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237989 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe1ecef000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8efe1ecef000.lock_unused_count=10 removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.05818 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.10664 s, 33.8 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg311-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg311-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 On MGS 192.168.203.111, active = nodemap.active=1 On MGS 192.168.203.111, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.800037 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110278 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107757 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222781 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293864 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243588 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223833 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307912 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327242 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe1ecef000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8efe1ecef000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314822 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386304 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319697 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386548 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393428 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362005 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397294 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408869 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155861 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1269 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131018 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167561 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136192 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181366 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142395 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142749 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153457 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194859 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154716 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157713 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160009 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16586 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172819 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189547 s, 5.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe1ecef000.lock_unused_count=4 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7246 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94606 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.111, TBF.id = On MGS 192.168.203.111, active = nodemap.active=0 PASS 77kj (185s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 06:50:07 (1788519007) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (12s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 06:50:18 (1788519018) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg311-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.69248 s, 284 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg311-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1165 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 06:51:18 (1788519078) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.586133 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804162 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854776 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144433 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811094 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172834 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189293 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206018 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175503 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190429 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140444 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188771 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674092 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195506 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247005 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259966 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288172 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe1ecef000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8efe1ecef000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114463 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594169 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116612 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784355 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110457 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037227 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101866 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128649 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683575 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074961 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046313 s, 22.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483866 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795847 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11597 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564901 s, 18.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe1ecef000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102718 s, 10.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg311-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg311-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.1279 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.71604 s, 18.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.588075 s, 28.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107576 s, 9.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102616 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106517 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971095 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138787 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108938 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104865 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147916 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213965 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220914 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283496 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286116 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8efe1ecef000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209561 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390103 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376398 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43582 s, 2.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8efe1ecef000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293234 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210096 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199275 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234935 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191119 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246086 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250494 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314619 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23101 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254154 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217317 s, 4.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334747 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334728 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241747 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262879 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355297 s, 3.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg311-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg311-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86725 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.29724 s, 19.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (99s) == sanityn test 77o: Changing rank should not panic ====== 06:52:57 (1788519177) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 06:53:11 (1788519191) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (113s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 06:55:04 (1788519304) mds.MDS.mdt.nrs_policies=tbf oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg311-client: oleg311-server: ssh exited with exit code 36 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg311-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg311-client: oleg311-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (36s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 06:55:41 (1788519341) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273396 kill md_thread xargs PASS 77r (57s) == sanityn test 77s: Check TBF LRU shrinker ============== 06:56:38 (1788519398) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 8 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 8 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (72s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 06:57:50 (1788519470) oleg311-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg311-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg311-client: oleg311-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 06:58:00 (1788519480) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 06:58:02.000000000 -0400 Modify: 2026-09-04 06:58:02.000000000 -0400 Change: 2026-09-04 06:58:02.000000000 -0400 Birth: 2026-09-04 06:58:03.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 06:58:19 (1788519499) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 06:58:23 (1788519503) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 06:58:26 (1788519506) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 06:58:30 (1788519510) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 06:58:35 (1788519515) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 06:58:39 (1788519519) - open/close 702 (time 1788519535.50 total 10.01 last 70.11) - open/close 1324 (time 1788519545.50 total 20.01 last 62.19) - open/close 2015 (time 1788519555.52 total 30.03 last 68.98) total: 2500 open/close in 37.78 seconds: 66.18 ops/second snapshot_time 1788519665.665577086 secs.nsecs start_time 1788519523.922314480 secs.nsecs elapsed_time 141.743262606 secs.nsecs open 2505 samples [usecs] 118 13950 3931419 10710590335 close 2505 samples [usecs] 135 7763 907970 722134988 mknod 2501 samples [usecs] 561 13663 3410578 8378709224 unlink 2500 samples [usecs] 760 101660 6269599 74445243631 mkdir 2 samples [usecs] 4197 7031 11228 67049770 rmdir 2 samples [usecs] 2003 25111 27114 634574330 rename 2500 samples [usecs] 464 57338 18847749 297138273779 getattr 3762 samples [usecs] 7 14134 323384 963410356 parallel_rename_file 2500 samples [usecs] 464 57338 18847749 297138273779 crossdir_rename 2500 samples [usecs] 464 57338 18847749 297138273779 readpage 24 samples [usecs] 5392 47814 567640 16528430760 PASS 81d (153s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 07:01:11 (1788519671) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7632 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7632 PASS 82 (8s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 07:01:19 (1788519679) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 07:01:23 (1788519683) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 07:01:36 (1788519696) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788519698 Finishing test test1 at 1788519703 PASS 85 (13s) == sanityn test 90: open/create and unlink striped directory ========================================================== 07:01:49 (1788519709) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 07:01:53 (1788519713) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 07:01:57 (1788519717) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 07:02:00 (1788519720) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5799 0x16a7 0x240000400 1 5762 0x1682 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5800 0x16a8 0x240000400 1 5763 0x1683 0x280000400 rc1=0 and rc2=0 PASS 93 (18s) == sanityn test 94: signal vs CP callback race =========== 07:02:18 (1788519738) osc.lustre-OST0000-osc-ffff8efe03500800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8efe1ecef000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8efe03500800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8efe1ecef000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.239735 s, 34.2 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.98541 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290133 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.133441 s, 30.7 kB/s osc.lustre-OST0000-osc-ffff8efe03500800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8efe1ecef000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8efe03500800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8efe1ecef000.idle_timeout=debug PASS 94 (17s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 07:02:35 (1788519755) llite.lustre-ffff8efe03500800.fast_read=0 llite.lustre-ffff8efe1ecef000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0957946 s, 21.9 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8efe03500800.fast_read=1 llite.lustre-ffff8efe1ecef000.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 07:02:56 (1788519776) llite.lustre-ffff8efe03500800.fast_read=0 llite.lustre-ffff8efe1ecef000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0120319 s, 1.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8efe03500800.fast_read=1 llite.lustre-ffff8efe1ecef000.fast_read=1 PASS 95b (15s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 07:03:11 (1788519791) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 07:03:15 (1788519795) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0166549 s, 246 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (9s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 07:03:24 (1788519804) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.34829 s, 6.0 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 07:03:33 (1788519813) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195379 s, 10.7 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 07:03:41 (1788519821) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0057035 s, 2.1 MB/s PASS 100e (9s) == sanityn test 101a: Discard DoM data on unlink ========= 07:03:50 (1788519830) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0181092 s, 226 kB/s PASS 101a (9s) == sanityn test 101b: Discard DoM data on rename ========= 07:03:59 (1788519839) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0202622 s, 202 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 07:04:07 (1788519847) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00618707 s, 662 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 07:04:15 (1788519855) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 07:04:24 (1788519864) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788519867 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788519870 Finishing test test23 at 1788519871 PASS 103 (16s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 07:04:40 (1788519880) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 07:04:44 (1788519884) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 07:05:07 (1788519907) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 07:05:12 (1788519912) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0229802 s, 44.6 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 07:05:14.000000000 -0400 Modify: 2026-09-04 07:05:14.000000000 -0400 Change: 2026-09-04 07:05:14.000000000 -0400 Birth: 2026-09-04 07:05:15.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938203 -rw-r--r-- 1 0 0 1788519915 1788519914 1788519914 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 07:05:14.000000000 -0400 Modify: 2026-09-04 07:05:14.000000000 -0400 Change: 2026-09-04 07:05:14.000000000 -0400 Birth: 2026-09-04 07:05:15.000000000 -0400 1788519914 1024 1 PASS 106b (10s) == sanityn test 106c: Verify statx attributes mask ======= 07:05:22 (1788519922) PASS 106c (9s) == sanityn test 107a: Basic grouplock conflict =========== 07:05:31 (1788519931) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.365835 s, 28.7 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5809 0x16b1 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 First grouplock blocks second one PASS 107a (11s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 07:05:42 (1788519942) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.469552 s, 22.3 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5770 0x168a 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7632 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 07:05:57 (1788519957) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0147582 s, 678 kB/s llite.lustre-ffff8efe03500800.hybrid_io=0 llite.lustre-ffff8efe1ecef000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.73989 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 07:06:10 (1788519970) 192.168.203.111@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre 192.168.203.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.111@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg311-client.virtnet /mnt/lustre (opts:) Stopping client oleg311-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing load_modules_local oleg311-server: kmemleak disabled oleg311-server: Loading modules from /home/green/git/lustre-release/lustre oleg311-server: detected 4 online CPUs by sysfs oleg311-server: MODOPTS_LIBCFS= oleg311-server: Force libcfs to create 2 CPU partitions oleg311-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre Starting client: oleg311-client.virtnet: -o user_xattr,flock 192.168.203.111@tcp:/lustre /mnt/lustre2 PASS 109 (47s) == sanityn test 110: do not grant another lock on resend ========================================================== 07:06:57 (1788520017) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 07:06:59.000000000 -0400 Modify: 2026-09-04 07:06:59.000000000 -0400 Change: 2026-09-04 07:06:59.000000000 -0400 Birth: 2026-09-04 07:07:00.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg311-client.virtnet oleg311-client.virtnet PASS 110 (71s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 07:08:08 (1788520088) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 07:08:12 (1788520092) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 07:08:17 (1788520097) lustre-OST0000-osc-ffff8efe11355800 active. lustre-OST0001-osc-ffff8efe11355800 active. lustre-OST0000-osc-ffff8efe11355800 active. lustre-OST0001-osc-ffff8efe11355800 active. MGC192.168.203.111@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 07:08:25 (1788520105) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 07:08:29 (1788520109) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 07:08:33 (1788520113) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 07:08:37 (1788520117) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0089728 s, 2.3 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0963243 s, 213 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 07:08:46 (1788520126) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 07:08:57 (1788520137) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1854 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1989 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.226 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1831 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2786 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1923 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2575 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1031 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2234 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3182 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2724 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3364 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3133 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2865 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3115 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3969 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4822 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4925 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg311-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8efe0338b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8efe0338b000.ost_server_uuid in IDLE state after 0 sec PASS 200 (83s) cleanup: ====================================================== == sanityn test complete, duration 10240 sec ============= 07:10:22 (1788520222) === sanityn: start cleanup 07:10:24 (1788520224) === Stopping clients: oleg311-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg311-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 07:15:38 (1788520538) ===