-----============= acceptance-small: sanityn ============----- Thu Sep 10 03:00:30 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg339-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 03:00:49 (1789023649) === Starting client oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Mount client oleg339-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Started clients oleg339-client.virtnet: 192.168.203.139@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg339-client.virtnet: executing check_config_client /mnt/lustre oleg339-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg339-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff94be06a1b800.idle_timeout=debug osc.lustre-OST0000-osc-ffff94be0727e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff94be06a1b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff94be0727e000.idle_timeout=debug disable quota as required oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 03:01:22 (1789023682) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418064 s, 25.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7614] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 03:01:24 (1789023684) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 03:01:32 (1789023692) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 03:01:39 (1789023699) -rw-r--r-- 1 root root 0 Sep 10 03:01 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 03:01:45 (1789023705) -rw-r--r-- 1 root root 0 Sep 10 03:01 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 03:01:51 (1789023711) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (5s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 03:01:56 (1789023716) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 03:02:04 (1789023724) total 1 -rwxrwxrwx 1 root root 0 Sep 10 03:01 f2a -rwxrwxrwx 1 root root 0 Sep 10 03:01 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 10 03:01 f2a -rwxrwxrwx 1 root root 0 Sep 10 03:01 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0208428 s, 24.6 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 03:02:11 (1789023731) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 03:02:14 (1789023734) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.639484 s, 3.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 03:02:22 (1789023742) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 03:02:30 (1789023750) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 03:02:38 (1789023758) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 03:02:45 (1789023765) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 03:02:52 (1789023772) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 03:02:58 (1789023778) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 03:03:04 (1789023784) PASS 9a (6s) == sanityn test 9b: append to striped sparse file ======== 03:03:10 (1789023790) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557461 s, 18.8 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0555244 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 03:03:16 (1789023796) 1+0 records in 1+0 records out 1 byte copied, 0.0114399 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0128008 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0176866 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0111996 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00943634 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0116995 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0105608 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00873183 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0163864 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0110157 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0336232 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0140167 s, 0.1 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 03:03:23 (1789023803) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0345814 s, 88.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0306649 s, 134 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000321401 s, 9.6 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 03:03:30 (1789023810) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 03:03:37 (1789023817) warning: '-runtime' deprecated, use '-t runtime' instead - link 319 (time 1789023830.16 total 10.02 last 31.85) - link 678 (time 1789023840.16 total 20.02 last 35.89) - link 1068 (time 1789023850.19 total 30.05 last 38.89) - link 1469 (time 1789023860.21 total 40.06 last 40.04) - link 1881 (time 1789023870.23 total 50.09 last 41.11) - link 2293 (time 1789023880.25 total 60.10 last 41.13) - link 2658 (time 1789023890.26 total 70.12 last 36.45) - link 3013 (time 1789023900.27 total 80.12 last 35.47) - link 3331 (time 1789023910.27 total 90.13 last 31.78) total: 3656 link in 100.01 seconds: 36.55 ops/second using seed 1337941489 running for 100 seconds - stat 5263 (time 1789023832 ; total 11 ; last 11) - stat 10000 (time 1789023840 ; total 19 ; last 8) - stat 15498 (time 1789023851 ; total 30 ; last 11) - stat 20000 (time 1789023858 ; total 37 ; last 7) - stat 26422 (time 1789023869 ; total 48 ; last 11) - stat 30000 (time 1789023878 ; total 57 ; last 9) - stat 32655 (time 1789023889 ; total 68 ; last 11) - stat 35598 (time 1789023900 ; total 79 ; last 11) - stat 38311 (time 1789023911 ; total 90 ; last 11) - stat 40000 (time 1789023917 ; total 96 ; last 6) total: 41120 stats in 100 seconds: 411.200012 stats/second - unlinked 0 (time 1789023871 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3656) error: No such file or directory total: 3655 unlinks in 90 seconds: 40.611111 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21427) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21429) - No such process PASS 12 (150s) == sanityn test 13: test directory page revocation ======= 03:06:07 (1789023967) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 03:06:15 (1789023975) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 22613 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 03:06:21 (1789023981) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 23192 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 03:06:28 (1789023988) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23772 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 03:06:35 (1789023995) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24354 Terminated $DIR1/$tdir/sleep 60 PASS 14c (9s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 03:06:44 (1789024004) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24937 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 03:06:53 (1789024013) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (46s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 03:07:40 (1789024060) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.493316 s, 17.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.373307 s, 22.5 MB/s Chance of close/open is 1 in 50 Seed set to 4064 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1789024065.475317 write 0x3b4c4d thru 0x3bfb4b (0xaeff bytes) 000003[0] 1789024065.530682 write 0x13d162 thru 0x1413cf (0x426e bytes) 000004[1] 1789024065.598987 mapread 0x3341bf thru 0x341574 (0xd3b6 bytes) truncating to largest ever: 0x77636c 000005[1] 1789024065.727056 trunc from 0x3bfb4c to 0x77636b (0x3b6820 bytes) 000006[1] 1789024065.789380 write 0x3f3a44 thru 0x402906 (0xeec3 bytes) 000008[1] 1789024065.857613 write 0x7ad7ef thru 0x7b3c8c (0x649e bytes) 000009[1] 1789024065.861182 read 0x5e0f75 thru 0x5ec482 (0xb50e bytes) 000010[0] 1789024065.877408 mapwrite 0x73b8e0 thru 0x745b50 (0xa271 bytes) 000014[0] 1789024066.091232 mapwrite 0x3d73e3 thru 0x3da1c3 (0x2de1 bytes) 000015[1] 1789024066.202844 mapread 0x6d6f94 thru 0x6e5e68 (0xeed5 bytes) 000016[0] 1789024066.299972 punch from 0x0b73e6 to 0x0bf58d (0x81a8 bytes) 000017[1] 1789024066.329998 punch from 0x71c6ad to 0x720ee8 (0x483c bytes) 000020[0] 1789024066.398162 read 0x49514e thru 0x4a15f4 (0xc4a7 bytes) 000022[0] 1789024066.450573 punch from 0x37d67b to 0x38b421 (0xdda7 bytes) 000023[0] 1789024066.456406 punch from 0x14cd02 to 0x15c354 (0xf653 bytes) 000024[1] 1789024066.495596 punch from 0x0e9c86 to 0x0f70fb (0xd476 bytes) 000025[0] 1789024066.534608 punch from 0x070062 to 0x073b43 (0x3ae2 bytes) 000026[1] 1789024066.591033 trunc from 0x7b3c8d to 0x29629b (0x51d9f1 bytes) 000027[0] 1789024066.643308 punch from 0x1e2d6f to 0x1eda47 (0xacd9 bytes) 000029[0] 1789024066.704253 trunc from 0x29629c to 0x4786c7 (0x1e242c bytes) 000033[1] 1789024066.851301 write 0x970249 thru 0x97ad2d (0xaae5 bytes) 000035[0] 1789024066.932379 mapread 0x7a35dc thru 0x7b1981 (0xe3a6 bytes) 000036[0] 1789024067.195932 read 0x4f1828 thru 0x4f3df3 (0x25cc bytes) 000037[1] 1789024067.238906 read 0x415448 thru 0x42215a (0xcd13 bytes) 000038[0] 1789024067.289888 read 0x640292 thru 0x641832 (0x15a1 bytes) 000041[1] 1789024067.336956 mapread 0x949a3e thru 0x951626 (0x7be9 bytes) 000042[1] 1789024067.357237 punch from 0x23ef19 to 0x2472b2 (0x839a bytes) truncating to largest ever: 0x7888c8 000043[0] 1789024067.379251 trunc from 0x97ad2e to 0x7888c7 (0x1f2466 bytes) 000045[0] 1789024067.443102 mapwrite 0x158b45 thru 0x163118 (0xa5d4 bytes) 000047[0] 1789024067.614154 trunc from 0x7888c8 to 0x2d06ed (0x4b81da bytes) 000048[1] 1789024067.684423 mapread 0x137e24 thru 0x13cf25 (0x5102 bytes) 000049[1] 1789024067.749529 punch from 0x25e05a to 0x2629e4 (0x498b bytes) 000053[0] 1789024067.931213 mapwrite 0x0f1983 thru 0x0fbbc3 (0xa241 bytes) 000054[0] 1789024068.085799 trunc from 0x2d06ee to 0x2dc720 (0xc033 bytes) 000055[1] 1789024068.135180 mapread 0x0d9c3b thru 0x0da30b (0x6d1 bytes) 000056[1] 1789024068.241089 mapwrite 0x277d24 thru 0x2799cd (0x1caa bytes) 000060[1] 1789024068.474926 trunc from 0x2dc721 to 0x424249 (0x147b29 bytes) 000062[0] 1789024068.506511 write 0x409486 thru 0x416778 (0xd2f3 bytes) 000063[1] 1789024068.551248 write 0x5e5f7a thru 0x5ede95 (0x7f1c bytes) 000064[0] 1789024068.604518 write 0x79677e thru 0x79a4c0 (0x3d43 bytes) 000065[0] 1789024068.659341 punch from 0x3b2328 to 0x3bffe8 (0xdcc1 bytes) 000066[0] 1789024068.687386 mapwrite 0x79adb4 thru 0x7a246e (0x76bb bytes) 000068[0] 1789024068.857541 punch from 0x525dc5 to 0x52d665 (0x78a1 bytes) 000071[1] 1789024068.909288 mapread 0x5c6f63 thru 0x5cd4c8 (0x6566 bytes) 000072[0] 1789024068.974250 punch from 0x072208 to 0x07d452 (0xb24b bytes) 000075[1] 1789024069.017873 punch from 0x1a3f4d to 0x1adda6 (0x9e5a bytes) 000077[1] 1789024069.044518 trunc from 0x7a246f to 0x388179 (0x41a2f5 bytes) 000084[0] 1789024069.306689 punch from 0x09f5a7 to 0x0a444d (0x4ea7 bytes) 000088[1] 1789024069.462753 trunc from 0x38817a to 0x095ba9 (0x2f25d0 bytes) 000090[1] 1789024069.552383 read 0x093e20 thru 0x095ba9 (0x1d8a bytes) truncating to largest ever: 0x7cfdb6 000091[0] 1789024069.576173 trunc from 0x095baa to 0x7cfdb5 (0x73a20c bytes) 000092[0] 1789024069.673062 write 0x988924 thru 0x9978e7 (0xefc4 bytes) 000093[0] 1789024069.678376 read 0x0f4e70 thru 0x10390b (0xea9c bytes) 000095[1] 1789024069.698013 mapread 0x8f2a75 thru 0x8fab83 (0x810f bytes) 000097[0] 1789024069.995819 mapread 0x28b2b6 thru 0x28f98a (0x46d5 bytes) 000099[1] 1789024070.041408 punch from 0x3d065c to 0x3df623 (0xefc8 bytes) 000100[1] 1789024070.106339 read 0x2da6a1 thru 0x2e6e6e (0xc7ce bytes) 000102[0] 1789024070.181939 mapwrite 0x4ca30f thru 0x4cc097 (0x1d89 bytes) 000105[0] 1789024070.343376 mapwrite 0x55d487 thru 0x56cbd4 (0xf74e bytes) 000107[1] 1789024070.498787 trunc from 0x9978e8 to 0x08d19f (0x90a748 bytes) 000112[0] 1789024070.676213 punch from 0x020308 to 0x025b36 (0x582f bytes) 000113[1] 1789024070.709274 mapwrite 0x06b115 thru 0x0708ca (0x57b6 bytes) 000115[1] 1789024070.829187 mapwrite 0x79299c thru 0x79920f (0x6874 bytes) 000117[1] 1789024070.924155 mapread 0x43edb8 thru 0x44aac8 (0xbd11 bytes) 000118[1] 1789024070.974036 mapread 0x1444c8 thru 0x14600f (0x1b48 bytes) 000119[0] 1789024071.015530 punch from 0x4942ac to 0x49bf13 (0x7c68 bytes) 000120[1] 1789024071.065917 mapwrite 0x9cacf9 thru 0x9cdad3 (0x2ddb bytes) 000122[0] 1789024071.241307 mapread 0x2944c4 thru 0x29df49 (0x9a86 bytes) 000123[1] 1789024071.343303 read 0x2613cf thru 0x26413b (0x2d6d bytes) 000124[0] 1789024071.356839 write 0x1640ee thru 0x1673f4 (0x3307 bytes) 000125[0] 1789024071.449568 mapwrite 0x2e2f51 thru 0x2f1bbc (0xec6c bytes) 000126[1] 1789024071.556593 trunc from 0x9cdad4 to 0x682d07 (0x34adcc bytes) 000127[0] 1789024071.592269 punch from 0x1272f8 to 0x127579 (0x282 bytes) 000128[1] 1789024071.623539 punch from 0x044351 to 0x04e25a (0x9f0a bytes) 000130[0] 1789024071.670642 read 0x20531e thru 0x2129e6 (0xd6c9 bytes) 000132[0] 1789024071.716943 trunc from 0x682d08 to 0x336ed4 (0x34be33 bytes) 000133[0] 1789024071.779880 write 0x97bbe3 thru 0x987ee4 (0xc302 bytes) 000135[1] 1789024071.834378 mapwrite 0x9f6aed thru 0x9f6b29 (0x3d bytes) 000136[0] 1789024071.967892 read 0x35e218 thru 0x360784 (0x256d bytes) 000137[0] 1789024071.990744 mapwrite 0x23ddef thru 0x2469b7 (0x8bc9 bytes) 000138[1] 1789024072.226482 trunc from 0x9f6b2a to 0x4231c1 (0x5d3968 bytes) 000139[0] 1789024072.294488 mapwrite 0x913769 thru 0x917a8b (0x4323 bytes) 000140[1] 1789024072.472931 punch from 0x8de861 to 0x8eb89f (0xd03f bytes) 000141[0] 1789024072.518244 punch from 0x25151e to 0x260e8c (0xf96f bytes) 000144[1] 1789024072.572336 read 0x5db5e3 thru 0x5e52cf (0x9ced bytes) 000146[1] 1789024072.620745 mapread 0x83c1d3 thru 0x8490ce (0xcefc bytes) 000147[0] 1789024072.704025 read 0x90de46 thru 0x90fd26 (0x1ee1 bytes) 000148[1] 1789024072.718956 mapwrite 0x886b34 thru 0x88fcbd (0x918a bytes) 000150[1] 1789024072.877924 write 0x82ac04 thru 0x832131 (0x752e bytes) 000151[0] 1789024072.907495 punch from 0x817ed5 to 0x826ab7 (0xebe3 bytes) 000153[1] 1789024072.936684 write 0x2c6a42 thru 0x2cd73e (0x6cfd bytes) 000154[1] 1789024072.976857 mapwrite 0x6538ae thru 0x6588db (0x502e bytes) 000155[0] 1789024073.110281 trunc from 0x917a8c to 0x355b00 (0x5c1f8b bytes) 000158[1] 1789024073.220981 punch from 0x00a8d6 to 0x00ca9f (0x21ca bytes) 000160[1] 1789024073.261359 mapread 0x2ccb8a thru 0x2d925b (0xc6d2 bytes) 000161[1] 1789024073.316238 read 0x164517 thru 0x16664b (0x2135 bytes) 000163[0] 1789024073.438212 read 0x1659a6 thru 0x168e04 (0x345f bytes) 000166[1] 1789024073.550939 punch from 0x252d0b to 0x25c9b3 (0x9ca9 bytes) 000168[1] 1789024073.628215 trunc from 0x355b01 to 0x31dba8 (0x37f58 bytes) 000172[0] 1789024073.816638 punch from 0x1895e0 to 0x18ebf4 (0x5615 bytes) 000173[1] 1789024073.880402 read 0x004dc7 thru 0x00dd16 (0x8f50 bytes) 000174[1] 1789024073.962101 punch from 0x02da79 to 0x03af2d (0xd4b5 bytes) 000175[1] 1789024074.010985 mapread 0x1e37fb thru 0x1ec058 (0x885e bytes) 000176[0] 1789024074.121469 write 0x536f55 thru 0x53d1f5 (0x62a1 bytes) 000177[1] 1789024074.158431 punch from 0x390b8c to 0x391baf (0x1024 bytes) 000178[0] 1789024074.216212 punch from 0x125700 to 0x12e1e1 (0x8ae2 bytes) 000180[1] 1789024074.294803 punch from 0x44a14f to 0x456146 (0xbff8 bytes) 000181[0] 1789024074.369342 read 0x06682a thru 0x06d74e (0x6f25 bytes) 000182[1] 1789024074.399348 mapread 0x1b7389 thru 0x1c6f96 (0xfc0e bytes) 000183[0] 1789024074.621581 punch from 0x161018 to 0x1628f6 (0x18df bytes) 000184[1] 1789024074.674478 read 0x0c40b2 thru 0x0d1105 (0xd054 bytes) 000188[0] 1789024074.735403 mapread 0x4367af thru 0x43b77b (0x4fcd bytes) 000191[1] 1789024074.793818 mapwrite 0x709b61 thru 0x7126f9 (0x8b99 bytes) 000195[0] 1789024074.977746 mapwrite 0x598ae7 thru 0x5a5fb0 (0xd4ca bytes) 000198[1] 1789024075.145145 mapwrite 0x4bd785 thru 0x4c06d7 (0x2f53 bytes) 000200[0] 1789024075.259221 mapwrite 0x6e964c thru 0x6f684e (0xd203 bytes) 000204[1] 1789024075.489935 punch from 0x2d8850 to 0x2dcedb (0x468c bytes) 000205[0] 1789024075.506873 write 0x42e272 thru 0x43d9ee (0xf77d bytes) 000207[0] 1789024075.548265 read 0x488466 thru 0x488b6f (0x70a bytes) 000208[1] 1789024075.577239 mapread 0x391e4a thru 0x398e98 (0x704f bytes) 000209[0] 1789024075.650604 punch from 0x2bee1b to 0x2c36fb (0x48e1 bytes) 000210[0] 1789024075.672129 read 0x4fc08e thru 0x50320d (0x7180 bytes) 000211[1] 1789024075.719211 write 0x1638ce thru 0x170613 (0xcd46 bytes) 000212[0] 1789024075.780553 mapread 0x34fc59 thru 0x3562c8 (0x6670 bytes) 000213[1] 1789024075.826554 mapread 0x2978fb thru 0x2a53f0 (0xdaf6 bytes) 000214[1] 1789024075.886382 trunc from 0x7126fa to 0x0be0b5 (0x654644 bytes) 000217[1] 1789024076.028978 trunc from 0x0be0b6 to 0x3e00e4 (0x32202f bytes) 000218[1] 1789024076.084295 trunc from 0x3e00e5 to 0x401e2e (0x21d4a bytes) 000219[0] 1789024076.110251 write 0x4d06eb thru 0x4d94ea (0x8e00 bytes) 000220[0] 1789024076.164103 mapread 0x1349b1 thru 0x135f5a (0x15aa bytes) 000221[0] 1789024076.206537 mapread 0x42a0ca thru 0x4308fb (0x6832 bytes) 000222[0] 1789024076.223837 read 0x197a7e thru 0x1a3c7d (0xc200 bytes) 000223[1] 1789024076.270621 mapread 0x1fc284 thru 0x2007d2 (0x454f bytes) 000225[1] 1789024076.323146 punch from 0x02516a to 0x029cad (0x4b44 bytes) 000227[1] 1789024076.369848 punch from 0x044de4 to 0x04af61 (0x617e bytes) 000230[1] 1789024076.423083 mapread 0x14963c thru 0x158ae8 (0xf4ad bytes) 000231[0] 1789024076.491439 mapread 0x2091ef thru 0x216773 (0xd585 bytes) 000233[0] 1789024076.599832 write 0x3e47fd thru 0x3e7a1c (0x3220 bytes) 000234[1] 1789024076.627875 mapread 0x0671e2 thru 0x0761e2 (0xf001 bytes) 000237[1] 1789024076.696179 read 0x2acdfd thru 0x2bc0b1 (0xf2b5 bytes) 000238[0] 1789024076.733791 read 0x00db5e thru 0x016294 (0x8737 bytes) 000240[0] 1789024076.771243 mapread 0x213e27 thru 0x214a30 (0xc0a bytes) 000241[1] 1789024076.783481 punch from 0x304f93 to 0x313153 (0xe1c1 bytes) 000242[0] 1789024076.827202 mapread 0x3d021b thru 0x3d5918 (0x56fe bytes) 000243[1] 1789024076.861701 mapwrite 0x53602e thru 0x5406e0 (0xa6b3 bytes) 000244[0] 1789024077.026051 read 0x126604 thru 0x12ddf3 (0x77f0 bytes) 000245[1] 1789024077.054265 mapwrite 0x67ca2c thru 0x685b4f (0x9124 bytes) 000250[0] 1789024077.256224 punch from 0x23b9a3 to 0x24a9ea (0xf048 bytes) 000252[0] 1789024077.327513 read 0x274763 thru 0x276e24 (0x26c2 bytes) 000254[1] 1789024077.379207 trunc from 0x685b50 to 0x72f693 (0xa9b44 bytes) 000255[1] 1789024077.441185 mapwrite 0x43d2f0 thru 0x44a7a5 (0xd4b6 bytes) 000259[1] 1789024077.617515 read 0x14483f thru 0x14850a (0x3ccc bytes) 000260[0] 1789024077.658155 read 0x2686e0 thru 0x276369 (0xdc8a bytes) 000261[0] 1789024077.700774 trunc from 0x72f694 to 0x0f32d4 (0x63c3bf bytes) 000262[1] 1789024077.763597 mapread 0x033226 thru 0x0336f7 (0x4d2 bytes) 000265[1] 1789024077.856496 mapread 0x09c56a thru 0x09d9a3 (0x143a bytes) 000267[0] 1789024077.919671 read 0x0ebfd6 thru 0x0f32d4 (0x72ff bytes) 000268[0] 1789024077.961593 mapread 0x07a891 thru 0x082d78 (0x84e8 bytes) 000269[1] 1789024077.964100 punch from 0x0269fe to 0x03331e (0xc921 bytes) 000270[1] 1789024078.004146 read 0x037599 thru 0x040339 (0x8da1 bytes) 000271[0] 1789024078.045551 write 0x0cc9bf thru 0x0cea34 (0x2076 bytes) 000272[0] 1789024078.101910 punch from 0x092684 to 0x096fa6 (0x4923 bytes) 000273[1] 1789024078.118969 write 0x7e388b thru 0x7f0361 (0xcad7 bytes) 000277[0] 1789024078.242679 mapwrite 0x5fb418 thru 0x5ff44d (0x4036 bytes) 000279[0] 1789024078.392440 punch from 0x42dc94 to 0x43a11f (0xc48c bytes) 000285[0] 1789024078.439246 mapread 0x1ec2da thru 0x1effc5 (0x3cec bytes) 000286[0] 1789024078.491324 mapwrite 0x5992a8 thru 0x59f207 (0x5f60 bytes) 000287[0] 1789024078.627542 mapread 0x4f95d2 thru 0x507dbd (0xe7ec bytes) 000288[0] 1789024078.714344 mapwrite 0x6e82ec thru 0x6f19e3 (0x96f8 bytes) 000290[1] 1789024078.818261 trunc from 0x7f0362 to 0x0871e1 (0x769180 bytes) 000292[1] 1789024078.914078 write 0x80330b thru 0x805274 (0x1f6a bytes) 000294[1] 1789024078.923526 mapread 0x7a16b3 thru 0x7a180c (0x15a bytes) 000295[1] 1789024079.005557 punch from 0x2f7c74 to 0x2fca1c (0x4da9 bytes) 000297[1] 1789024079.052109 mapread 0x61bcde thru 0x61eba1 (0x2ec4 bytes) 000298[1] 1789024079.095256 mapread 0x01299f thru 0x016ca3 (0x4305 bytes) 000302[0] 1789024079.202807 trunc from 0x805275 to 0x6889ec (0x17c888 bytes) 000303[0] 1789024079.264318 mapwrite 0x272329 thru 0x27475e (0x2436 bytes) 000304[1] 1789024079.421788 mapwrite 0x9ee35e thru 0x9fc99e (0xe641 bytes) 000308[0] 1789024079.630099 mapwrite 0x4ca404 thru 0x4cc506 (0x2103 bytes) 000312[0] 1789024079.811689 trunc from 0x9fc99f to 0x19b6d7 (0x8612c7 bytes) 000314[1] 1789024079.897514 write 0x170004 thru 0x170d3e (0xd3b bytes) 000316[0] 1789024080.001034 mapread 0x046133 thru 0x051ca3 (0xbb71 bytes) 000319[0] 1789024080.213469 mapread 0x008d97 thru 0x00f74a (0x69b4 bytes) 000321[1] 1789024080.272885 read 0x055491 thru 0x05f61a (0xa18a bytes) 000326[0] 1789024080.336284 mapwrite 0x7893d9 thru 0x796d61 (0xd989 bytes) 000327[1] 1789024080.507491 mapwrite 0x9c67eb thru 0x9c96db (0x2ef1 bytes) 000328[1] 1789024080.872424 write 0x7b4410 thru 0x7b6fb1 (0x2ba2 bytes) 000330[0] 1789024080.937648 write 0x224d57 thru 0x22cdb7 (0x8061 bytes) 000331[1] 1789024080.980345 trunc from 0x9c96dc to 0x23bd09 (0x78d9d2 bytes) 000332[1] 1789024081.057819 punch from 0x03bc76 to 0x049cb6 (0xe041 bytes) 000333[0] 1789024081.085518 mapwrite 0x3473b1 thru 0x352293 (0xaee3 bytes) 000336[1] 1789024081.344494 mapread 0x03cf11 thru 0x04714a (0xa23a bytes) 000337[0] 1789024081.522177 mapwrite 0x492e84 thru 0x49e361 (0xb4de bytes) 000338[1] 1789024081.787558 mapread 0x33d690 thru 0x3437f9 (0x616a bytes) 000339[0] 1789024081.867262 mapread 0x2762b8 thru 0x2770fa (0xe43 bytes) 000340[0] 1789024081.896264 mapread 0x14421b thru 0x14d78f (0x9575 bytes) 000343[0] 1789024081.972650 read 0x122581 thru 0x132246 (0xfcc6 bytes) truncating to largest ever: 0x9595da 000344[0] 1789024081.982694 trunc from 0x49e362 to 0x9595d9 (0x4bb278 bytes) 000346[1] 1789024082.087620 mapwrite 0x8959e7 thru 0x8a3d79 (0xe393 bytes) 000347[0] 1789024082.345161 write 0x3945f0 thru 0x39781c (0x322d bytes) 000350[0] 1789024082.407604 write 0x28b103 thru 0x2969f9 (0xb8f7 bytes) 000351[1] 1789024082.421298 punch from 0x857d5c to 0x864e5c (0xd101 bytes) 000352[0] 1789024082.482010 read 0x91cb39 thru 0x928069 (0xb531 bytes) 000353[0] 1789024082.501966 read 0x4f77ac thru 0x503458 (0xbcad bytes) 000354[1] 1789024082.522590 punch from 0x1869b7 to 0x18c24c (0x5896 bytes) 000355[0] 1789024082.554308 punch from 0x0c7c7f to 0x0d0786 (0x8b08 bytes) 000356[1] 1789024082.571141 read 0x3e885f thru 0x3eddf9 (0x559b bytes) 000357[0] 1789024082.606432 mapwrite 0x3471f5 thru 0x355a45 (0xe851 bytes) 000358[0] 1789024082.714369 trunc from 0x9595da to 0x1b582c (0x7a3dad bytes) 000359[0] 1789024082.770289 write 0x6b212d thru 0x6ba9be (0x8892 bytes) 000362[0] 1789024082.808279 trunc from 0x6ba9bf to 0x233699 (0x487325 bytes) 000364[0] 1789024082.929368 trunc from 0x23369a to 0x2c0fef (0x8d956 bytes) 000365[0] 1789024082.958441 mapread 0x1ddd10 thru 0x1ea86b (0xcb5c bytes) 000366[0] 1789024082.989990 write 0x33a875 thru 0x33cadb (0x2267 bytes) 000367[1] 1789024083.018245 read 0x0cdf40 thru 0x0d13fe (0x34bf bytes) 000368[1] 1789024083.113208 trunc from 0x33cadc to 0x3d8b92 (0x9c0b7 bytes) 000373[1] 1789024083.270275 mapwrite 0x828fd8 thru 0x8315ca (0x85f3 bytes) 000377[1] 1789024083.512110 read 0x12a74d thru 0x13735e (0xcc12 bytes) 000380[1] 1789024083.569142 punch from 0x6e8632 to 0x6f7ffd (0xf9cc bytes) 000381[0] 1789024083.573498 mapwrite 0x844c43 thru 0x849fcc (0x538a bytes) 000382[0] 1789024083.728015 write 0x94160f thru 0x94da70 (0xc462 bytes) 000383[1] 1789024083.734593 mapwrite 0x0d2f07 thru 0x0d30e7 (0x1e1 bytes) 000384[1] 1789024083.857131 read 0x36491d thru 0x369c95 (0x5379 bytes) 000385[0] 1789024083.882489 trunc from 0x94da71 to 0x34d23d (0x600833 bytes) 000386[0] 1789024083.953094 mapread 0x065597 thru 0x06e8d9 (0x9343 bytes) 000388[1] 1789024084.067113 read 0x2a18ff thru 0x2a429b (0x299d bytes) 000390[1] 1789024084.198454 write 0x998429 thru 0x99a1a1 (0x1d79 bytes) 000395[1] 1789024084.341548 write 0x0a5d25 thru 0x0a7ea9 (0x2185 bytes) 000396[1] 1789024084.377555 mapwrite 0x9465e6 thru 0x95025f (0x9c7a bytes) 000397[1] 1789024084.532768 punch from 0x67da61 to 0x68b54e (0xdaee bytes) 000398[0] 1789024084.559758 punch from 0x0843c9 to 0x090cca (0xc902 bytes) 000403[1] 1789024084.714569 trunc from 0x99a1a2 to 0x8b9102 (0xe109f bytes) 000405[1] 1789024084.775010 read 0x15e6ad thru 0x15f7f8 (0x114c bytes) 000407[1] 1789024084.824310 read 0x8b1d21 thru 0x8b4554 (0x2834 bytes) 000408[1] 1789024084.835504 trunc from 0x8b9103 to 0x703955 (0x1b57ad bytes) 000409[0] 1789024084.871849 read 0x6605b2 thru 0x666739 (0x6188 bytes) 000411[0] 1789024084.957589 write 0x648943 thru 0x64c148 (0x3806 bytes) 000414[0] 1789024085.049831 write 0x35cbf3 thru 0x369b2d (0xcf3b bytes) truncating to largest ever: 0x9be5bd 000418[1] 1789024085.174627 trunc from 0x703956 to 0x9be5bc (0x2bac67 bytes) 000419[0] 1789024085.247765 punch from 0x9a08dc to 0x9a4d11 (0x4436 bytes) 000420[0] 1789024085.283628 read 0x1a5cf0 thru 0x1b38ec (0xdbfd bytes) 000421[0] 1789024085.308775 write 0x6b6b02 thru 0x6bb09c (0x459b bytes) 000422[1] 1789024085.355232 read 0x607a83 thru 0x60f5ab (0x7b29 bytes) 000423[1] 1789024085.387431 punch from 0x233f88 to 0x239452 (0x54cb bytes) 000426[0] 1789024085.421218 punch from 0x2fdfba to 0x2ffae6 (0x1b2d bytes) 000427[0] 1789024085.461014 mapwrite 0x7ee4ad thru 0x7f92d4 (0xae28 bytes) 000429[1] 1789024085.615154 mapread 0x407f24 thru 0x40ef46 (0x7023 bytes) 000431[1] 1789024085.728835 mapread 0x2facc1 thru 0x302315 (0x7655 bytes) 000432[0] 1789024085.830290 mapread 0x4e381c thru 0x4f0053 (0xc838 bytes) 000433[1] 1789024085.892068 write 0x07d2ff thru 0x0877cd (0xa4cf bytes) 000435[1] 1789024085.984401 write 0x7f48f7 thru 0x8046ab (0xfdb5 bytes) 000439[1] 1789024086.187452 mapwrite 0x1ddf0e thru 0x1e3c04 (0x5cf7 bytes) 000440[1] 1789024086.351383 mapwrite 0x4b2b51 thru 0x4bd347 (0xa7f7 bytes) 000441[0] 1789024086.513552 read 0x13f286 thru 0x1420de (0x2e59 bytes) 000444[0] 1789024086.538890 write 0x8c383e thru 0x8cd8d9 (0xa09c bytes) 000445[1] 1789024086.582370 mapread 0x69cb6a thru 0x6ab985 (0xee1c bytes) 000447[1] 1789024086.693941 mapread 0x5550bc thru 0x556ef9 (0x1e3e bytes) 000448[0] 1789024086.751253 read 0x6347be thru 0x63767f (0x2ec2 bytes) 000453[1] 1789024086.848546 mapread 0x96136e thru 0x96f48c (0xe11f bytes) 000455[0] 1789024086.924452 punch from 0x816135 to 0x824abc (0xe988 bytes) 000457[0] 1789024086.950696 mapwrite 0x208b90 thru 0x21779d (0xec0e bytes) 000458[1] 1789024087.079421 trunc from 0x9be5bd to 0x4900f4 (0x52e4c8 bytes) 000459[1] 1789024087.195798 write 0x366b09 thru 0x37037f (0x9877 bytes) 000462[1] 1789024087.276158 trunc from 0x4900f5 to 0x929d47 (0x499c53 bytes) 000464[0] 1789024087.371531 mapread 0x3f37f1 thru 0x3f79cb (0x41db bytes) 000465[1] 1789024087.475604 punch from 0x54f076 to 0x553983 (0x490e bytes) 000466[1] 1789024087.505104 write 0x74289f thru 0x74c551 (0x9cb3 bytes) 000468[0] 1789024087.556133 punch from 0x2051a6 to 0x20b277 (0x60d2 bytes) 000469[1] 1789024087.598565 write 0x517c6d thru 0x520c58 (0x8fec bytes) 000470[1] 1789024087.640820 mapread 0x0680be thru 0x06ee9a (0x6ddd bytes) 000472[1] 1789024087.755919 read 0x20788c thru 0x211ae0 (0xa255 bytes) 000477[0] 1789024087.836031 write 0x7b41ed thru 0x7b9a06 (0x581a bytes) 000480[0] 1789024087.929292 punch from 0x7b58c9 to 0x7b7352 (0x1a8a bytes) 000483[1] 1789024087.980397 write 0x22f10a thru 0x23cdc1 (0xdcb8 bytes) 000484[1] 1789024088.018043 punch from 0x64f881 to 0x654f09 (0x5689 bytes) 000485[0] 1789024088.071271 mapwrite 0x45ae8d thru 0x464609 (0x977d bytes) 000488[0] 1789024088.299338 trunc from 0x929d48 to 0x57c800 (0x3ad547 bytes) 000489[1] 1789024088.346555 mapwrite 0x51837c thru 0x523252 (0xaed7 bytes) 000492[0] 1789024088.684534 read 0x120430 thru 0x126d6f (0x6940 bytes) 000493[1] 1789024088.751970 trunc from 0x57c801 to 0x81b3bb (0x29ebbb bytes) 000496[0] 1789024088.850544 write 0x9bc246 thru 0x9cad60 (0xeb1b bytes) 000497[0] 1789024088.876208 punch from 0x7556c9 to 0x75bb0f (0x6447 bytes) 000498[1] 1789024088.915178 read 0x4ba411 thru 0x4baf31 (0xb21 bytes) 000500[0] 1789024088.993699 mapread 0x966387 thru 0x96ed09 (0x8983 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4089 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole skipping zero size read skipping zero length punch hole skipping zero size read 000005[1] 1789024089.966539 mapwrite 0x3d8000 thru 0x3e37bd (0xb7be bytes) truncating to largest ever: 0x415fa4 000006[0] 1789024090.191107 trunc from 0x3e37be to 0x415fa3 (0x327e6 bytes) truncating to largest ever: 0x4d7bbe 000007[0] 1789024090.248448 trunc from 0x415fa4 to 0x4d7bbd (0xc1c1a bytes) 000008[1] 1789024090.273348 read 0x1da000 thru 0x1e9e77 (0xfe78 bytes) 000009[0] 1789024090.312162 mapread 0x376000 thru 0x379e83 (0x3e84 bytes) 000010[0] 1789024090.345367 mapread 0x255000 thru 0x260e2d (0xbe2e bytes) 000011[0] 1789024090.402491 read 0x125000 thru 0x12c97f (0x7980 bytes) 000014[0] 1789024090.443440 mapread 0x48c000 thru 0x49afb6 (0xefb7 bytes) 000015[0] 1789024090.518193 trunc from 0x4d7bbe to 0x01c4df (0x4bb6de bytes) 000016[0] 1789024090.611628 write 0x0c4000 thru 0x0cff3f (0xbf40 bytes) 000018[1] 1789024090.657314 write 0x0e0000 thru 0x0efa86 (0xfa87 bytes) 000019[0] 1789024090.697265 read 0x086000 thru 0x08a98d (0x498e bytes) truncating to largest ever: 0x938e4c 000020[0] 1789024090.751254 trunc from 0x0efa87 to 0x938e4b (0x8493c5 bytes) 000021[0] 1789024090.859879 trunc from 0x938e4c to 0x145c57 (0x7f31f4 bytes) 000022[1] 1789024090.885343 punch from 0x03fe2a to 0x042b4c (0x2d23 bytes) 000023[0] 1789024090.936551 trunc from 0x145c58 to 0x4e7f3d (0x3a22e6 bytes) 000024[1] 1789024090.975743 mapwrite 0x08b000 thru 0x0968bb (0xb8bc bytes) 000025[0] 1789024091.117805 mapwrite 0x01f000 thru 0x027367 (0x8368 bytes) 000026[0] 1789024091.466741 punch from 0x4bf40f to 0x4c6c8e (0x7880 bytes) 000027[1] 1789024091.478542 read 0x31b000 thru 0x329984 (0xe985 bytes) 000028[1] 1789024091.515072 read 0x339000 thru 0x3473b7 (0xe3b8 bytes) 000029[0] 1789024091.526809 mapwrite 0x009000 thru 0x00ec6a (0x5c6b bytes) 000030[1] 1789024091.708155 mapwrite 0x836000 thru 0x83aae4 (0x4ae5 bytes) 000033[1] 1789024091.965373 write 0x17c000 thru 0x1879ed (0xb9ee bytes) 000035[1] 1789024092.028285 mapread 0x271000 thru 0x271a9b (0xa9c bytes) 000037[0] 1789024092.058632 mapread 0x4d5000 thru 0x4e427f (0xf280 bytes) 000039[1] 1789024092.236816 write 0x676000 thru 0x67996b (0x396c bytes) 000040[1] 1789024092.324187 read 0x54b000 thru 0x552383 (0x7384 bytes) 000041[0] 1789024092.343016 punch from 0x228fae to 0x2347a5 (0xb7f8 bytes) 000042[1] 1789024092.374559 punch from 0x0452c5 to 0x04f5ef (0xa32b bytes) 000044[1] 1789024092.435521 write 0x75e000 thru 0x76c863 (0xe864 bytes) 000046[1] 1789024092.460578 mapread 0x546000 thru 0x55219c (0xc19d bytes) 000047[1] 1789024092.505485 read 0x135000 thru 0x13b047 (0x6048 bytes) 000050[1] 1789024092.525683 trunc from 0x83aae5 to 0x529dcf (0x310d15 bytes) 000052[0] 1789024092.555372 write 0x4ac000 thru 0x4b6604 (0xa605 bytes) 000053[1] 1789024092.603229 read 0x4c8000 thru 0x4ccaf8 (0x4af9 bytes) 000056[0] 1789024092.677196 trunc from 0x529dd0 to 0x7cb6ac (0x2a18dd bytes) 000057[1] 1789024092.778163 trunc from 0x7cb6ad to 0x3b6544 (0x415168 bytes) 000059[1] 1789024092.837522 punch from 0x25a5f3 to 0x267f46 (0xd954 bytes) 000061[0] 1789024092.860465 mapread 0x212000 thru 0x219c3f (0x7c40 bytes) 000062[0] 1789024092.954798 mapwrite 0x751000 thru 0x755ca1 (0x4ca2 bytes) 000063[0] 1789024093.199758 mapwrite 0x035000 thru 0x03fdcf (0xadd0 bytes) 000065[0] 1789024093.354539 read 0x2d6000 thru 0x2de972 (0x8973 bytes) 000066[0] 1789024093.380535 read 0x4cc000 thru 0x4cd4a4 (0x14a5 bytes) 000067[1] 1789024093.421614 write 0x995000 thru 0x996925 (0x1926 bytes) 000068[0] 1789024093.480433 read 0x95e000 thru 0x96c51b (0xe51c bytes) 000072[1] 1789024093.538670 mapwrite 0x06d000 thru 0x06e07b (0x107c bytes) 000076[0] 1789024093.685609 mapwrite 0x74f000 thru 0x75d36d (0xe36e bytes) 000077[1] 1789024093.802105 trunc from 0x996926 to 0x27000e (0x726917 bytes) 000078[0] 1789024093.895086 read 0x1dc000 thru 0x1e18af (0x58b0 bytes) 000080[1] 1789024093.984564 write 0x64c000 thru 0x651c4f (0x5c50 bytes) 000083[0] 1789024094.053812 mapread 0x3f0000 thru 0x3f0448 (0x449 bytes) 000084[0] 1789024094.210027 punch from 0x0c93a7 to 0x0d52fd (0xbf57 bytes) 000086[1] 1789024094.259450 mapread 0x050000 thru 0x055c74 (0x5c75 bytes) 000092[1] 1789024094.307066 write 0x0c4000 thru 0x0d2d1d (0xed1e bytes) 000094[1] 1789024094.343015 trunc from 0x651c50 to 0x80f771 (0x1bdb22 bytes) 000095[0] 1789024094.381484 trunc from 0x80f772 to 0x467ffc (0x3a7775 bytes) 000096[1] 1789024094.431285 trunc from 0x467ffd to 0x19ea84 (0x2c9578 bytes) 000097[1] 1789024094.463614 mapwrite 0x590000 thru 0x59ccfd (0xccfe bytes) 000099[1] 1789024094.590337 read 0x257000 thru 0x264183 (0xd184 bytes) 000100[1] 1789024094.603215 write 0x994000 thru 0x99d716 (0x9717 bytes) 000102[1] 1789024094.636652 read 0x4f6000 thru 0x4fbe9c (0x5e9d bytes) truncating to largest ever: 0x964661 000103[0] 1789024094.686347 trunc from 0x99d717 to 0x964660 (0x390b6 bytes) 000104[0] 1789024094.742971 mapread 0x87a000 thru 0x87e6c0 (0x46c1 bytes) 000106[0] 1789024094.832526 mapread 0x5d7000 thru 0x5dd7b0 (0x67b1 bytes) 000109[1] 1789024094.909402 mapread 0x49d000 thru 0x4ac710 (0xf711 bytes) 000110[1] 1789024094.951529 trunc from 0x964661 to 0x362cce (0x601992 bytes) 000111[1] 1789024094.997291 punch from 0x14d535 to 0x14f38c (0x1e58 bytes) 000114[1] 1789024095.011606 punch from 0x04148f to 0x04d98f (0xc501 bytes) 000115[1] 1789024095.034177 read 0x228000 thru 0x22e710 (0x6711 bytes) 000116[1] 1789024095.125274 punch from 0x0273cb to 0x028ff6 (0x1c2c bytes) 000119[1] 1789024095.176091 mapread 0x2c5000 thru 0x2cc608 (0x7609 bytes) 000122[1] 1789024095.192275 mapread 0x038000 thru 0x038c08 (0xc09 bytes) 000124[0] 1789024095.193741 mapread 0x259000 thru 0x265af6 (0xcaf7 bytes) 000125[0] 1789024095.292486 read 0x0a5000 thru 0x0b2379 (0xd37a bytes) 000126[0] 1789024095.352462 trunc from 0x362ccf to 0x514f6c (0x1b229e bytes) 000127[1] 1789024095.461918 trunc from 0x514f6d to 0x1a47b7 (0x3707b5 bytes) 000128[0] 1789024095.527246 trunc from 0x1a47b8 to 0x286504 (0xe1d4d bytes) 000130[1] 1789024095.557344 mapwrite 0x5cd000 thru 0x5cd932 (0x933 bytes) 000132[0] 1789024095.697637 write 0x4a2000 thru 0x4b0a6b (0xea6c bytes) 000133[0] 1789024095.779464 mapread 0x18a000 thru 0x197a48 (0xda49 bytes) 000134[0] 1789024095.946232 mapwrite 0x426000 thru 0x4265ca (0x5cb bytes) 000136[0] 1789024096.104109 mapread 0x347000 thru 0x35315a (0xc15b bytes) 000138[1] 1789024096.295749 mapwrite 0x28a000 thru 0x292737 (0x8738 bytes) 000140[0] 1789024096.540069 write 0x877000 thru 0x880835 (0x9836 bytes) 000141[0] 1789024096.616037 read 0x092000 thru 0x0a04e0 (0xe4e1 bytes) 000142[0] 1789024096.671413 trunc from 0x880836 to 0x1861b9 (0x6fa67c bytes) 000144[1] 1789024096.734272 mapread 0x067000 thru 0x06a4ef (0x34f0 bytes) 000145[1] 1789024096.786319 punch from 0x045bd6 to 0x04f945 (0x9d70 bytes) 000146[1] 1789024096.800329 trunc from 0x1861ba to 0x85c972 (0x6d67b9 bytes) 000147[0] 1789024096.832129 trunc from 0x85c973 to 0x678d1a (0x1e3c58 bytes) 000148[1] 1789024096.929366 punch from 0x4e08f0 to 0x4e7f26 (0x7637 bytes) 000149[0] 1789024096.992494 read 0x5be000 thru 0x5c07cb (0x27cc bytes) 000151[1] 1789024097.035435 mapwrite 0x0e5000 thru 0x0f26b4 (0xd6b5 bytes) 000153[1] 1789024097.249987 write 0x573000 thru 0x57cec0 (0x9ec1 bytes) 000154[1] 1789024097.297605 mapwrite 0x092000 thru 0x094bd9 (0x2bda bytes) 000156[1] 1789024097.489118 mapwrite 0x504000 thru 0x50e73f (0xa740 bytes) 000157[1] 1789024097.774954 write 0x2ed000 thru 0x2f2476 (0x5477 bytes) 000160[0] 1789024097.798369 trunc from 0x678d1b to 0x5571ff (0x121b1b bytes) 000162[1] 1789024097.881083 mapread 0x3ab000 thru 0x3b5f47 (0xaf48 bytes) 000163[0] 1789024097.990131 punch from 0x118292 to 0x123ca5 (0xba14 bytes) 000165[0] 1789024098.057022 read 0x4c7000 thru 0x4d4ab0 (0xdab1 bytes) 000167[1] 1789024098.120323 mapread 0x3e1000 thru 0x3ebebb (0xaebc bytes) 000168[1] 1789024098.269112 write 0x11f000 thru 0x12caad (0xdaae bytes) 000170[1] 1789024098.295146 trunc from 0x557200 to 0x24724b (0x30ffb4 bytes) 000171[1] 1789024098.319504 read 0x17b000 thru 0x182e2c (0x7e2d bytes) 000173[1] 1789024098.431985 punch from 0x0b1bce to 0x0b852e (0x6961 bytes) 000175[0] 1789024098.441467 punch from 0x071fe9 to 0x07fded (0xde05 bytes) 000176[1] 1789024098.488113 write 0x257000 thru 0x261756 (0xa757 bytes) 000179[0] 1789024098.529246 read 0x12e000 thru 0x13afc4 (0xcfc5 bytes) 000180[0] 1789024098.608501 mapread 0x161000 thru 0x167b4f (0x6b50 bytes) 000182[0] 1789024098.609685 read 0x22f000 thru 0x23ca07 (0xda08 bytes) 000183[1] 1789024098.610652 mapread 0x100000 thru 0x1001b5 (0x1b6 bytes) 000184[0] 1789024098.624447 trunc from 0x261757 to 0x8d1b41 (0x6703eb bytes) 000185[1] 1789024098.692405 punch from 0x89424e to 0x89bc28 (0x79db bytes) 000188[1] 1789024098.760228 read 0x4e2000 thru 0x4e9a0b (0x7a0c bytes) 000190[1] 1789024098.809365 mapwrite 0x62e000 thru 0x636447 (0x8448 bytes) 000192[1] 1789024099.059591 read 0x484000 thru 0x485f2a (0x1f2b bytes) 000193[0] 1789024099.079229 mapwrite 0x032000 thru 0x03a4e6 (0x84e7 bytes) 000194[0] 1789024099.305408 mapwrite 0x03a000 thru 0x0449b1 (0xa9b2 bytes) 000195[0] 1789024099.441962 punch from 0x174410 to 0x17a42b (0x601c bytes) 000196[0] 1789024099.486200 trunc from 0x8d1b42 to 0x56903c (0x368b05 bytes) 000199[1] 1789024099.536645 trunc from 0x56903d to 0x1cb68a (0x39d9b2 bytes) 000201[1] 1789024099.611236 mapread 0x16f000 thru 0x175c38 (0x6c39 bytes) 000203[1] 1789024099.635009 mapread 0x1b7000 thru 0x1c1f4d (0xaf4e bytes) 000206[1] 1789024099.718539 trunc from 0x1cb68b to 0x403e6a (0x2387e0 bytes) 000208[1] 1789024099.763229 write 0x087000 thru 0x08eed4 (0x7ed5 bytes) 000211[0] 1789024099.784656 read 0x093000 thru 0x0989ca (0x59cb bytes) 000214[1] 1789024099.828211 mapwrite 0x6f1000 thru 0x6f3f96 (0x2f97 bytes) 000215[0] 1789024100.024441 trunc from 0x6f3f97 to 0x0787a0 (0x67b7f6 bytes) 000218[0] 1789024100.133531 read 0x009000 thru 0x016c38 (0xdc39 bytes) 000219[1] 1789024100.164663 punch from 0x003bf4 to 0x006591 (0x299e bytes) 000224[1] 1789024100.219293 mapread 0x06a000 thru 0x0733e6 (0x93e7 bytes) 000225[0] 1789024100.248976 trunc from 0x0787a1 to 0x181ce8 (0x109548 bytes) 000226[1] 1789024100.351689 trunc from 0x181ce9 to 0x45f994 (0x2ddcac bytes) 000227[1] 1789024100.420009 punch from 0x3e1cc4 to 0x3eb6f4 (0x9a31 bytes) 000230[0] 1789024100.486165 mapread 0x0ed000 thru 0x0f09dd (0x39de bytes) 000232[0] 1789024100.599638 write 0x44d000 thru 0x44db9a (0xb9b bytes) 000233[1] 1789024100.641687 mapwrite 0x645000 thru 0x6452c8 (0x2c9 bytes) 000234[0] 1789024100.849442 mapread 0x2cb000 thru 0x2dafc2 (0xffc3 bytes) 000235[0] 1789024101.058273 mapread 0x01d000 thru 0x0262a0 (0x92a1 bytes) 000237[1] 1789024101.137572 punch from 0x391f28 to 0x39b1b8 (0x9291 bytes) 000238[0] 1789024101.172061 read 0x2f5000 thru 0x2f5e7d (0xe7e bytes) 000239[0] 1789024101.196423 mapread 0x2cd000 thru 0x2d2f98 (0x5f99 bytes) 000240[0] 1789024101.216889 write 0x157000 thru 0x166f88 (0xff89 bytes) 000241[0] 1789024101.249630 mapread 0x1e2000 thru 0x1efbc1 (0xdbc2 bytes) 000242[0] 1789024101.290642 read 0x4ed000 thru 0x4f6b45 (0x9b46 bytes) 000243[1] 1789024101.311695 read 0x081000 thru 0x086d98 (0x5d99 bytes) 000245[0] 1789024101.355602 read 0x121000 thru 0x125eb0 (0x4eb1 bytes) 000246[0] 1789024101.369344 trunc from 0x6452c9 to 0x6e759f (0xa22d7 bytes) 000249[1] 1789024101.427700 mapwrite 0x526000 thru 0x52c68a (0x668b bytes) 000252[0] 1789024101.572355 trunc from 0x6e75a0 to 0x5ffb8a (0xe7a15 bytes) 000253[1] 1789024101.617336 read 0x091000 thru 0x09730f (0x6310 bytes) 000255[0] 1789024101.656091 read 0x47a000 thru 0x47d972 (0x3973 bytes) 000256[0] 1789024101.676415 mapwrite 0x5b6000 thru 0x5c5ef8 (0xfef9 bytes) 000258[0] 1789024101.911959 write 0x9bf000 thru 0x9c5ecd (0x6ece bytes) 000260[1] 1789024101.964203 punch from 0x57c462 to 0x58acbd (0xe85c bytes) 000263[0] 1789024102.024330 read 0x85a000 thru 0x867362 (0xd363 bytes) 000266[0] 1789024102.068881 mapwrite 0x635000 thru 0x637b98 (0x2b99 bytes) 000267[0] 1789024102.228753 write 0x3a9000 thru 0x3b5e66 (0xce67 bytes) 000269[1] 1789024102.273437 write 0x62a000 thru 0x639e57 (0xfe58 bytes) 000270[0] 1789024102.366112 mapwrite 0x9f4000 thru 0x9f8b72 (0x4b73 bytes) 000272[0] 1789024102.589573 mapwrite 0x4a6000 thru 0x4b1ec1 (0xbec2 bytes) 000274[1] 1789024102.764801 punch from 0x674dbb to 0x683703 (0xe949 bytes) 000275[1] 1789024102.812277 trunc from 0x9f8b73 to 0x891ec6 (0x166cac bytes) 000276[0] 1789024102.850435 trunc from 0x891ec7 to 0x83616a (0x5bd5c bytes) 000277[0] 1789024102.904475 write 0x24a000 thru 0x24ef46 (0x4f47 bytes) 000280[1] 1789024102.970002 mapwrite 0x019000 thru 0x026256 (0xd257 bytes) 000281[1] 1789024103.089467 trunc from 0x83616b to 0x01696b (0x81f7ff bytes) 000282[0] 1789024103.143663 trunc from 0x01696c to 0x0c9c2c (0xb32c1 bytes) 000283[1] 1789024103.197449 mapread 0x0ac000 thru 0x0b10e3 (0x50e4 bytes) 000286[0] 1789024103.259276 write 0x39b000 thru 0x3aaeb2 (0xfeb3 bytes) 000287[0] 1789024103.283267 read 0x053000 thru 0x058f1f (0x5f20 bytes) 000290[1] 1789024103.358408 write 0x803000 thru 0x80eb4c (0xbb4d bytes) 000291[1] 1789024103.386633 read 0x30b000 thru 0x3161f0 (0xb1f1 bytes) 000292[1] 1789024103.401049 mapread 0x40d000 thru 0x413b52 (0x6b53 bytes) 000293[0] 1789024103.421639 punch from 0x76495b to 0x771149 (0xc7ef bytes) 000296[0] 1789024103.462456 mapwrite 0x6da000 thru 0x6dc3c8 (0x23c9 bytes) 000297[1] 1789024103.584580 read 0x6b1000 thru 0x6b589c (0x489d bytes) 000299[0] 1789024103.653925 read 0x640000 thru 0x64c2cb (0xc2cc bytes) 000301[0] 1789024103.688784 mapwrite 0x347000 thru 0x34fa3f (0x8a40 bytes) 000303[1] 1789024103.923808 trunc from 0x80eb4d to 0x3410a9 (0x4cdaa3 bytes) 000306[1] 1789024104.002303 mapread 0x1ea000 thru 0x1f21fc (0x81fd bytes) 000307[0] 1789024104.030256 read 0x2c7000 thru 0x2c9557 (0x2558 bytes) 000311[1] 1789024104.146617 mapread 0x0f3000 thru 0x1009c5 (0xd9c6 bytes) 000313[0] 1789024104.184335 punch from 0x17d863 to 0x185291 (0x7a2f bytes) 000317[0] 1789024104.235223 mapread 0x1ef000 thru 0x1f540a (0x640b bytes) 000318[0] 1789024104.312151 mapread 0x318000 thru 0x32796c (0xf96d bytes) 000319[1] 1789024104.489445 mapread 0x2ce000 thru 0x2cf6d8 (0x16d9 bytes) 000321[0] 1789024104.558484 read 0x091000 thru 0x09f003 (0xe004 bytes) 000322[0] 1789024104.645072 mapwrite 0x83f000 thru 0x84b704 (0xc705 bytes) 000326[1] 1789024104.941607 read 0x135000 thru 0x142f90 (0xdf91 bytes) 000328[0] 1789024105.138543 mapread 0x612000 thru 0x61b1e3 (0x91e4 bytes) 000329[1] 1789024105.210994 trunc from 0x84b705 to 0x7a7909 (0xa3dfb bytes) 000330[1] 1789024105.288045 punch from 0x0a0a20 to 0x0aea2d (0xe00e bytes) 000331[0] 1789024105.314114 read 0x4cd000 thru 0x4d5800 (0x8801 bytes) 000334[1] 1789024105.343472 mapwrite 0x307000 thru 0x30719e (0x19f bytes) 000335[1] 1789024105.467275 trunc from 0x7a790a to 0x50cd25 (0x29abe4 bytes) 000336[0] 1789024105.513505 mapread 0x49d000 thru 0x4a5ee6 (0x8ee7 bytes) 000337[1] 1789024105.600255 read 0x4e1000 thru 0x4ec2c5 (0xb2c6 bytes) 000338[0] 1789024105.619542 read 0x114000 thru 0x11ae48 (0x6e49 bytes) 000340[1] 1789024105.633427 mapwrite 0x6cd000 thru 0x6ce6f6 (0x16f7 bytes) 000343[0] 1789024105.762283 mapwrite 0x1d6000 thru 0x1e566e (0xf66f bytes) 000347[1] 1789024105.919675 read 0x4fc000 thru 0x4fe4a2 (0x24a3 bytes) 000348[1] 1789024105.953553 mapread 0x23b000 thru 0x243f88 (0x8f89 bytes) 000352[1] 1789024105.981682 write 0x921000 thru 0x929f46 (0x8f47 bytes) 000354[1] 1789024106.019320 read 0x8f9000 thru 0x90481b (0xb81c bytes) 000357[1] 1789024106.029477 mapwrite 0x15f000 thru 0x168b8c (0x9b8d bytes) 000358[1] 1789024106.112557 trunc from 0x929f47 to 0x5043e8 (0x425b5e bytes) 000360[0] 1789024106.148661 punch from 0x3d5959 to 0x3d8ba9 (0x3251 bytes) 000362[1] 1789024106.212578 write 0x74c000 thru 0x7509ee (0x49ef bytes) 000364[0] 1789024106.274313 write 0x84c000 thru 0x85b8f9 (0xf8fa bytes) 000365[1] 1789024106.309992 mapread 0x3db000 thru 0x3e9837 (0xe838 bytes) 000366[0] 1789024106.381868 trunc from 0x85b8fa to 0x7b1376 (0xaa583 bytes) 000367[1] 1789024106.429555 write 0x740000 thru 0x74def1 (0xdef2 bytes) 000368[0] 1789024106.464080 mapread 0x2b4000 thru 0x2bf84a (0xb84b bytes) 000369[0] 1789024106.532529 punch from 0x6416b3 to 0x64bfe1 (0xa92f bytes) 000370[0] 1789024106.583086 mapwrite 0x75b000 thru 0x764a86 (0x9a87 bytes) 000371[0] 1789024106.746640 mapwrite 0x50e000 thru 0x51a7de (0xc7df bytes) 000372[1] 1789024106.911125 mapwrite 0x0fb000 thru 0x1083c2 (0xd3c3 bytes) 000373[0] 1789024107.066419 trunc from 0x7b1377 to 0x558555 (0x258e21 bytes) 000374[0] 1789024107.135853 mapread 0x46b000 thru 0x4718c1 (0x68c2 bytes) 000375[1] 1789024107.162863 trunc from 0x558556 to 0x22b775 (0x32cde0 bytes) 000376[1] 1789024107.244519 punch from 0x1cc129 to 0x1d2253 (0x612b bytes) 000379[0] 1789024107.298673 punch from 0x0d03f9 to 0x0d1477 (0x107f bytes) 000381[1] 1789024107.353150 trunc from 0x22b776 to 0x0360c8 (0x1f56ad bytes) 000382[0] 1789024107.434295 trunc from 0x0360c9 to 0x361a9c (0x32b9d4 bytes) 000383[0] 1789024107.485296 trunc from 0x361a9d to 0x58996e (0x227ed2 bytes) 000387[1] 1789024107.603445 mapwrite 0x2d6000 thru 0x2e5ef4 (0xfef5 bytes) 000388[1] 1789024107.765500 trunc from 0x58996f to 0x576a19 (0x12f55 bytes) 000391[1] 1789024107.822595 read 0x3a4000 thru 0x3a4cd0 (0xcd1 bytes) 000392[1] 1789024107.850563 mapread 0x3a4000 thru 0x3b39e8 (0xf9e9 bytes) 000393[1] 1789024107.874981 mapread 0x369000 thru 0x36ec8e (0x5c8f bytes) 000394[1] 1789024107.882345 punch from 0x00c22a to 0x012212 (0x5fe9 bytes) 000395[1] 1789024107.887262 mapread 0x428000 thru 0x42de84 (0x5e85 bytes) 000396[0] 1789024107.931239 mapwrite 0x621000 thru 0x621069 (0x6a bytes) 000399[0] 1789024108.121994 punch from 0x12a249 to 0x133e65 (0x9c1d bytes) 000400[0] 1789024108.149121 mapread 0x61b000 thru 0x620d49 (0x5d4a bytes) 000402[1] 1789024108.167807 mapread 0x5d2000 thru 0x5db0e8 (0x90e9 bytes) 000404[0] 1789024108.236091 mapread 0x5f4000 thru 0x5fc32c (0x832d bytes) 000406[1] 1789024108.251891 punch from 0x0ded70 to 0x0ed623 (0xe8b4 bytes) 000410[0] 1789024108.294252 mapread 0x3b3000 thru 0x3b521e (0x221f bytes) 000411[1] 1789024108.333549 trunc from 0x62106a to 0x60a044 (0x17025 bytes) 000412[1] 1789024108.419179 read 0x128000 thru 0x12ac18 (0x2c19 bytes) 000416[0] 1789024108.444407 mapread 0x3f8000 thru 0x3fd9e5 (0x59e6 bytes) 000418[0] 1789024108.499593 read 0x5c5000 thru 0x5c80c4 (0x30c5 bytes) 000419[1] 1789024108.511236 read 0x5f1000 thru 0x5f45f3 (0x35f4 bytes) 000420[1] 1789024108.525067 punch from 0x1e2b4b to 0x1f17f2 (0xeca8 bytes) 000421[0] 1789024108.560076 mapread 0x043000 thru 0x04b3e9 (0x83ea bytes) 000422[0] 1789024108.602489 mapwrite 0x520000 thru 0x529cda (0x9cdb bytes) 000423[0] 1789024108.675986 punch from 0x391697 to 0x39c99b (0xb305 bytes) 000424[0] 1789024108.694300 write 0x338000 thru 0x339fe6 (0x1fe7 bytes) 000429[1] 1789024108.804572 punch from 0x20de7c to 0x21aada (0xcc5f bytes) 000432[0] 1789024108.852048 read_OD 0x0e1000 thru 0x0e5fff (0x5000 bytes) 000433[0] 1789024108.877896 mapwrite 0x22d000 thru 0x22fd76 (0x2d77 bytes) 000436[1] 1789024108.962783 punch from 0x4671a2 to 0x470607 (0x9466 bytes) 000437[0] 1789024108.981599 read_OD 0x016000 thru 0x019fff (0x4000 bytes) 000439[0] 1789024109.010766 write_OD 0x35e000 thru 0x36bfff (0xe000 bytes) 000440[1] 1789024109.113375 read 0x0a4000 thru 0x0b28ce (0xe8cf bytes) 000442[0] 1789024109.130145 read_OD 0x35c000 thru 0x365fff (0xa000 bytes) 000443[0] 1789024109.148578 read_OD 0x266000 thru 0x270fff (0xb000 bytes) 000444[1] 1789024109.161468 punch from 0x3e96e2 to 0x3ec942 (0x3261 bytes) 000445[1] 1789024109.205666 write 0x686000 thru 0x68f417 (0x9418 bytes) 000447[0] 1789024109.233117 trunc from 0x68f418 to 0x441151 (0x24e2c6 bytes) 000450[0] 1789024109.338522 mapread 0x3c3000 thru 0x3ca456 (0x7457 bytes) 000452[1] 1789024109.379129 read 0x17c000 thru 0x1875f0 (0xb5f1 bytes) 000455[0] 1789024109.408309 punch from 0x171895 to 0x17526c (0x39d8 bytes) 000457[0] 1789024109.440198 punch from 0x4000be to 0x408439 (0x837c bytes) 000458[0] 1789024109.445490 punch from 0x2a748e to 0x2b013e (0x8cb1 bytes) 000459[1] 1789024109.465321 punch from 0x2743f1 to 0x27c3c7 (0x7fd7 bytes) 000460[0] 1789024109.497096 punch from 0x34edad to 0x354029 (0x527d bytes) 000462[0] 1789024109.548623 trunc from 0x441152 to 0x27db07 (0x1c364a bytes) 000463[1] 1789024109.572274 punch from 0x25e88d to 0x264a25 (0x6199 bytes) 000465[0] 1789024109.608493 read_OD 0x15e000 thru 0x16afff (0xd000 bytes) 000467[1] 1789024109.637676 mapread 0x251000 thru 0x252eb8 (0x1eb9 bytes) 000468[1] 1789024109.648264 mapread 0x249000 thru 0x2503e5 (0x73e6 bytes) 000469[0] 1789024109.661170 read_OD 0x1f0000 thru 0x1fcfff (0xd000 bytes) 000470[1] 1789024109.669412 punch from 0x079df2 to 0x07fa2c (0x5c3b bytes) 000471[1] 1789024109.742088 mapread 0x233000 thru 0x23acfc (0x7cfd bytes) 000473[0] 1789024109.774528 write_OD 0x716000 thru 0x720fff (0xb000 bytes) 000474[1] 1789024109.842526 mapread 0x305000 thru 0x30eb8c (0x9b8d bytes) 000475[1] 1789024109.884158 mapread 0x3d7000 thru 0x3d8b44 (0x1b45 bytes) 000476[0] 1789024109.894965 mapwrite 0x2c5000 thru 0x2ca0ef (0x50f0 bytes) 000478[0] 1789024110.068063 write_OD 0x02c000 thru 0x031fff (0x6000 bytes) 000479[0] 1789024110.192589 punch from 0x2101e7 to 0x21801d (0x7e37 bytes) 000480[0] 1789024110.265613 read_OD 0x2d9000 thru 0x2e4fff (0xc000 bytes) 000484[1] 1789024110.310322 mapwrite 0x8fe000 thru 0x9082f1 (0xa2f2 bytes) 000486[0] 1789024110.449964 read_OD 0x3f6000 thru 0x402fff (0xd000 bytes) 000488[0] 1789024110.482416 trunc from 0x9082f2 to 0x346472 (0x5c1e7f bytes) 000489[1] 1789024110.529165 mapread 0x0fe000 thru 0x10a9ee (0xc9ef bytes) 000491[1] 1789024110.592021 read 0x069000 thru 0x06ba63 (0x2a64 bytes) 000492[0] 1789024110.640894 mapread 0x27d000 thru 0x2832b6 (0x62b7 bytes) 000494[0] 1789024110.662210 trunc from 0x346473 to 0x6fd969 (0x3b74f7 bytes) 000495[1] 1789024110.708292 read 0x318000 thru 0x320d42 (0x8d43 bytes) 000496[1] 1789024110.819602 read 0x613000 thru 0x620455 (0xd456 bytes) 000498[0] 1789024110.839497 trunc from 0x6fd96a to 0x606924 (0xf7045 bytes) 000499[1] 1789024110.897236 write 0x468000 thru 0x47186b (0x986c bytes) All operations completed A-OK! PASS 16a (55s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 03:08:35 (1789024115) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.393935 s, 21.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.368619 s, 22.8 MB/s Chance of close/open is 1 in 50 Seed set to 4119 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x3ece2 000001[0] 1789024120.129894 trunc from 00000000 to 0x03ece1 (0x3ece2 bytes) truncating to largest ever: 0x7ff686 000002[0] 1789024120.161432 trunc from 0x03ece2 to 0x7ff685 (0x7c09a4 bytes) 000004[0] 1789024120.238530 punch from 0x75426e to 0x754c27 (0x9ba bytes) 000005[0] 1789024120.255394 punch from 0x5e618c to 0x5e70c6 (0xf3b bytes) 000006[1] 1789024120.260465 mapwrite 0x22527d thru 0x226918 (0x169c bytes) 000008[0] 1789024120.421480 read 0x05c63e thru 0x05c70c (0xcf bytes) 000009[1] 1789024120.442037 read 0x47ced2 thru 0x47d62d (0x75c bytes) 000010[1] 1789024120.497991 read 0x5671ce thru 0x568447 (0x127a bytes) 000012[0] 1789024120.517607 mapwrite 0x81983a thru 0x81a281 (0xa48 bytes) 000013[1] 1789024120.693546 mapwrite 0x28eb01 thru 0x28f26b (0x76b bytes) 000014[0] 1789024120.859606 trunc from 0x81a282 to 0x2a6532 (0x573d4f bytes) 000015[1] 1789024120.923087 mapwrite 0x102d5e thru 0x1048d1 (0x1b74 bytes) 000016[1] 1789024121.072518 write 0x79f3af thru 0x7a0c7c (0x18ce bytes) 000019[0] 1789024121.117225 punch from 0x380bb3 to 0x381e3d (0x128b bytes) 000020[0] 1789024121.155218 write 0x40a599 thru 0x40b040 (0xaa8 bytes) 000021[1] 1789024121.211201 read 0x281826 thru 0x282618 (0xdf3 bytes) 000024[1] 1789024121.319981 read 0x5ad478 thru 0x5adce0 (0x869 bytes) 000025[0] 1789024121.384821 mapwrite 0x115e20 thru 0x117df3 (0x1fd4 bytes) truncating to largest ever: 0x8db0f4 000028[0] 1789024121.545067 trunc from 0x7a0c7d to 0x8db0f3 (0x13a477 bytes) 000030[1] 1789024121.637441 trunc from 0x8db0f4 to 0x1a8eca (0x732229 bytes) 000031[1] 1789024121.768184 trunc from 0x1a8ecb to 0x770d3a (0x5c7e70 bytes) 000032[0] 1789024121.850476 trunc from 0x770d3b to 0x750716 (0x20624 bytes) 000035[0] 1789024121.999932 mapwrite 0x53906d thru 0x5391bd (0x151 bytes) 000036[1] 1789024122.104760 read 0x3ddd61 thru 0x3dedd0 (0x1070 bytes) 000037[1] 1789024122.137172 mapread 0x5f50ca thru 0x5f6595 (0x14cc bytes) 000038[1] 1789024122.197680 read 0x03355e thru 0x03532f (0x1dd2 bytes) 000040[0] 1789024122.205034 write 0x5d1c78 thru 0x5d201f (0x3a8 bytes) 000043[1] 1789024122.302569 mapwrite 0x03d473 thru 0x03dc4e (0x7dc bytes) 000044[0] 1789024122.468278 punch from 0x29fa7d to 0x2a0ae7 (0x106b bytes) truncating to largest ever: 0x8e28ff 000045[0] 1789024122.520585 trunc from 0x750717 to 0x8e28fe (0x1921e8 bytes) 000046[1] 1789024122.569512 mapwrite 0x55aea1 thru 0x55c435 (0x1595 bytes) 000047[1] 1789024122.693239 punch from 0x14509b to 0x145e8f (0xdf5 bytes) 000048[1] 1789024122.738596 trunc from 0x8e28ff to 0x050673 (0x89228b bytes) 000049[0] 1789024122.785245 read 0x02b294 thru 0x02d058 (0x1dc5 bytes) 000050[1] 1789024122.833741 mapread 0x003ffb thru 0x004c90 (0xc96 bytes) 000051[0] 1789024122.885333 punch from 0x037965 to 0x038aae (0x114a bytes) 000052[1] 1789024122.914052 punch from 0x03dba9 to 0x03fadc (0x1f34 bytes) 000055[0] 1789024123.000089 punch from 0x0174f8 to 0x017f63 (0xa6c bytes) 000059[1] 1789024123.165167 trunc from 0x050674 to 0x4757dd (0x42516a bytes) 000060[1] 1789024123.233458 trunc from 0x4757de to 0x314f8d (0x160850 bytes) 000061[0] 1789024123.305438 mapread 0x2a5ecc thru 0x2a6a0d (0xb42 bytes) 000063[0] 1789024123.375594 mapwrite 0x53c562 thru 0x53cfc4 (0xa63 bytes) 000064[0] 1789024123.606007 mapwrite 0x873045 thru 0x874910 (0x18cc bytes) 000065[0] 1789024123.713433 mapwrite 0x12be09 thru 0x12d960 (0x1b58 bytes) 000067[1] 1789024123.778305 mapread 0x6e77a2 thru 0x6e8416 (0xc75 bytes) 000069[1] 1789024123.843931 mapread 0x06537d thru 0x06633d (0xfc1 bytes) 000070[0] 1789024123.872163 write 0x6c54a7 thru 0x6c5a40 (0x59a bytes) 000071[0] 1789024123.927623 mapread 0x1c2fba thru 0x1c43df (0x1426 bytes) 000072[0] 1789024123.949735 punch from 0x0338f7 to 0x0358e6 (0x1ff0 bytes) 000073[0] 1789024124.008360 read 0x7caaf6 thru 0x7cb941 (0xe4c bytes) 000075[0] 1789024124.039030 mapwrite 0x418cd6 thru 0x4191a1 (0x4cc bytes) 000077[0] 1789024124.113092 punch from 0x1a045a to 0x1a08b5 (0x45c bytes) 000078[1] 1789024124.149024 trunc from 0x874911 to 0x2a5c15 (0x5cecfb bytes) 000079[0] 1789024124.267575 punch from 0x1e8b5c to 0x1e9290 (0x735 bytes) 000081[0] 1789024124.352542 punch from 0x0bdf07 to 0x0be9bd (0xab7 bytes) 000082[1] 1789024124.393152 write 0x5c87f5 thru 0x5c9cc6 (0x14d2 bytes) 000086[0] 1789024124.520968 trunc from 0x5c9cc7 to 0x22ffe9 (0x399cdd bytes) 000087[0] 1789024124.554333 trunc from 0x22ffea to 0x0bbe33 (0x1741b6 bytes) 000089[0] 1789024124.570624 trunc from 0x0bbe34 to 0x2e811d (0x22c2ea bytes) 000090[1] 1789024124.625057 mapwrite 0x45c110 thru 0x45db4b (0x1a3c bytes) 000091[1] 1789024124.716406 trunc from 0x45db4c to 0x00ffd4 (0x44db77 bytes) 000092[1] 1789024124.782488 punch from 0x00ae76 to 0x00b57a (0x705 bytes) 000093[0] 1789024124.833209 read 0x008604 thru 0x009c57 (0x1654 bytes) 000094[1] 1789024124.875829 trunc from 0x00ffd5 to 0x23e4d2 (0x22e4fe bytes) 000096[1] 1789024124.964647 punch from 0x0e197a to 0x0e235c (0x9e3 bytes) 000097[1] 1789024124.986630 read 0x0fe4e5 thru 0x0ffbcf (0x16eb bytes) 000098[1] 1789024125.108510 trunc from 0x23e4d3 to 0x5261c8 (0x2e7cf6 bytes) 000099[0] 1789024125.185773 trunc from 0x5261c9 to 0x3861f9 (0x19ffcf bytes) 000100[0] 1789024125.269290 trunc from 0x3861fa to 0x09dd2f (0x2e84ca bytes) 000101[1] 1789024125.352546 trunc from 0x09dd30 to 0x6d78f3 (0x639bc4 bytes) 000102[0] 1789024125.405307 write 0x8f5707 thru 0x8f6b4e (0x1448 bytes) 000104[0] 1789024125.480266 mapwrite 0x705e4b thru 0x706f30 (0x10e6 bytes) 000105[1] 1789024125.784596 trunc from 0x8f6b4f to 0x552dc5 (0x3a3d89 bytes) 000106[1] 1789024125.915920 write 0x0f2efd thru 0x0f3a36 (0xb3a bytes) 000107[1] 1789024126.008232 trunc from 0x552dc6 to 0x4ec192 (0x66c33 bytes) 000109[0] 1789024126.047457 punch from 0x4a6449 to 0x4a6cb4 (0x86c bytes) 000114[1] 1789024126.196278 mapread 0x2195f0 thru 0x21abd1 (0x15e2 bytes) 000115[1] 1789024126.236055 mapwrite 0x3c5298 thru 0x3c5ca2 (0xa0b bytes) 000116[1] 1789024126.305140 read 0x103ea5 thru 0x104402 (0x55e bytes) 000117[0] 1789024126.324142 mapwrite 0x3e95e6 thru 0x3ea5bc (0xfd7 bytes) 000118[1] 1789024126.400586 mapread 0x4193bb thru 0x419e2b (0xa71 bytes) 000119[0] 1789024126.447697 mapread 0x001ce6 thru 0x001f93 (0x2ae bytes) 000121[0] 1789024126.488391 punch from 0x0d8397 to 0x0d8ae5 (0x74f bytes) 000122[1] 1789024126.507449 mapwrite 0x2839b4 thru 0x2844a7 (0xaf4 bytes) truncating to largest ever: 0x990d70 000128[1] 1789024126.655127 trunc from 0x4ec193 to 0x990d6f (0x4a4bdd bytes) 000130[0] 1789024126.762280 read 0x20367a thru 0x203d23 (0x6aa bytes) 000132[1] 1789024126.806550 mapread 0x938e43 thru 0x93a5e2 (0x17a0 bytes) 000133[0] 1789024126.849306 trunc from 0x990d70 to 0x1c9d66 (0x7c7009 bytes) 000135[0] 1789024126.911908 mapread 0x174ba1 thru 0x174c6d (0xcd bytes) 000137[1] 1789024126.972520 punch from 0x067543 to 0x0678be (0x37c bytes) 000138[1] 1789024127.011395 punch from 0x18295d to 0x183394 (0xa38 bytes) 000139[1] 1789024127.035216 mapwrite 0x8f5ac4 thru 0x8f6fd9 (0x1516 bytes) 000140[1] 1789024127.112365 mapread 0x48c6c2 thru 0x48e220 (0x1b5f bytes) 000142[1] 1789024127.135062 punch from 0x566390 to 0x567812 (0x1483 bytes) 000145[0] 1789024127.169663 read 0x23abeb thru 0x23ae0c (0x222 bytes) 000151[1] 1789024127.206422 punch from 0x226b0a to 0x227ec9 (0x13c0 bytes) 000153[1] 1789024127.228193 trunc from 0x8f6fda to 0x11e46c (0x7d8b6d bytes) 000154[0] 1789024127.332239 mapwrite 0x7cbcdf thru 0x7cd8af (0x1bd1 bytes) 000155[1] 1789024127.603895 punch from 0x31fc29 to 0x321b86 (0x1f5e bytes) 000158[0] 1789024127.752574 punch from 0x75226a to 0x7536cb (0x1462 bytes) 000160[0] 1789024127.836911 mapwrite 0x956299 thru 0x958182 (0x1eea bytes) 000162[1] 1789024128.124399 read 0x714ddb thru 0x714ec3 (0xe9 bytes) 000164[0] 1789024128.160243 read 0x18f45e thru 0x19062c (0x11cf bytes) 000165[0] 1789024128.168498 mapwrite 0x600877 thru 0x6012d9 (0xa63 bytes) 000166[1] 1789024128.339640 read 0x20af3d thru 0x20bb78 (0xc3c bytes) 000167[0] 1789024128.438914 read 0x5fc3e2 thru 0x5fcfef (0xc0e bytes) 000168[1] 1789024128.495546 read 0x4314af thru 0x431d5f (0x8b1 bytes) 000169[0] 1789024128.552296 trunc from 0x958183 to 0x368fc8 (0x5ef1ba bytes) 000170[0] 1789024128.653370 mapread 0x20024a thru 0x201b6a (0x1921 bytes) 000171[1] 1789024128.756635 write 0x6a43bb thru 0x6a5682 (0x12c8 bytes) 000172[1] 1789024128.820861 punch from 0x185e1a to 0x187708 (0x18ef bytes) 000173[0] 1789024128.925277 write 0x6ac75a thru 0x6ad462 (0xd09 bytes) 000174[1] 1789024128.981122 punch from 0x07fada to 0x0800f8 (0x61f bytes) 000175[0] 1789024128.999934 write 0x6cb2f4 thru 0x6ccb43 (0x1850 bytes) 000176[0] 1789024129.014533 mapread 0x3a8fe3 thru 0x3aaa93 (0x1ab1 bytes) 000178[0] 1789024129.065234 punch from 0x162b0a to 0x163bf6 (0x10ed bytes) 000180[1] 1789024129.103541 write 0x5a1581 thru 0x5a28f5 (0x1375 bytes) 000181[0] 1789024129.178081 mapread 0x53987b thru 0x539d00 (0x486 bytes) 000183[0] 1789024129.246453 mapwrite 0x2df465 thru 0x2e04ca (0x1066 bytes) 000184[1] 1789024129.331167 mapread 0x07c97f thru 0x07e744 (0x1dc6 bytes) 000186[1] 1789024129.396276 trunc from 0x6ccb44 to 0x458afb (0x274048 bytes) 000187[0] 1789024129.460402 read 0x172a70 thru 0x173e93 (0x1424 bytes) 000188[1] 1789024129.522964 trunc from 0x458afc to 0x6b8882 (0x25fd87 bytes) 000189[0] 1789024129.577167 read 0x234fc4 thru 0x236729 (0x1766 bytes) 000191[1] 1789024129.613090 mapwrite 0x8096b4 thru 0x80a061 (0x9ae bytes) 000192[1] 1789024129.808191 write 0x7a634c thru 0x7a6629 (0x2de bytes) 000193[0] 1789024129.831569 punch from 0x5991bd to 0x5999db (0x81f bytes) 000196[0] 1789024129.950966 trunc from 0x80a062 to 0x0bf046 (0x74b01b bytes) 000198[1] 1789024130.084194 read 0x031bc2 thru 0x0333f2 (0x1831 bytes) 000199[1] 1789024130.202284 mapwrite 0x88e66f thru 0x88ec6f (0x601 bytes) 000200[0] 1789024130.494012 trunc from 0x88ec70 to 0x0bc9a5 (0x7d22ca bytes) 000203[0] 1789024130.566446 mapread 0x06f98a thru 0x07137f (0x19f6 bytes) 000206[0] 1789024130.660517 read 0x07500a thru 0x075c44 (0xc3b bytes) 000207[1] 1789024130.661508 trunc from 0x0bc9a6 to 0x35ceb8 (0x2a0513 bytes) 000210[1] 1789024130.758838 mapread 0x0be0b4 thru 0x0bed87 (0xcd4 bytes) 000211[1] 1789024130.808265 mapwrite 0x8ef683 thru 0x8efecc (0x84a bytes) 000213[0] 1789024131.012157 trunc from 0x8efecd to 0x270dd5 (0x67f0f7 bytes) 000217[0] 1789024131.159308 mapwrite 0x1bb51b thru 0x1bd12f (0x1c15 bytes) 000218[0] 1789024131.272784 mapread 0x036367 thru 0x03698e (0x628 bytes) 000220[1] 1789024131.345454 mapwrite 0x29ad7a thru 0x29b895 (0xb1c bytes) 000221[1] 1789024131.488200 write 0x6a98e8 thru 0x6aab25 (0x123e bytes) 000223[0] 1789024131.553278 mapread 0x185d19 thru 0x1862a0 (0x588 bytes) 000225[1] 1789024131.634737 mapwrite 0x8d8e91 thru 0x8dac7b (0x1deb bytes) 000226[1] 1789024131.725590 trunc from 0x8dac7c to 0x936b4e (0x5bed3 bytes) 000229[0] 1789024131.808432 punch from 0x021c08 to 0x02320a (0x1603 bytes) 000230[0] 1789024131.874581 punch from 0x146e5f to 0x1479c2 (0xb64 bytes) 000231[1] 1789024131.905242 write 0x213a38 thru 0x214f62 (0x152b bytes) 000232[0] 1789024131.933169 read 0x332fe8 thru 0x334033 (0x104c bytes) 000234[1] 1789024131.972570 punch from 0x7b79ba to 0x7b8f81 (0x15c8 bytes) 000235[0] 1789024131.977892 read 0x85ac41 thru 0x85ba38 (0xdf8 bytes) 000236[0] 1789024132.011781 punch from 0x925d9d to 0x9275f1 (0x1855 bytes) 000237[1] 1789024132.051384 punch from 0x76ac97 to 0x76be1c (0x1186 bytes) 000241[1] 1789024132.125315 mapread 0x2d04f0 thru 0x2d12bd (0xdce bytes) 000243[0] 1789024132.182465 mapwrite 0x2bdc35 thru 0x2bfaab (0x1e77 bytes) 000244[0] 1789024132.294274 read 0x4ad093 thru 0x4ad789 (0x6f7 bytes) 000245[0] 1789024132.321065 write 0x99e65c thru 0x9a033d (0x1ce2 bytes) 000246[0] 1789024132.341031 write 0x19fbff thru 0x1a104a (0x144c bytes) 000247[1] 1789024132.354212 mapwrite 0x81a8c4 thru 0x81b493 (0xbd0 bytes) 000248[1] 1789024132.489886 read 0x534301 thru 0x534bd5 (0x8d5 bytes) 000250[1] 1789024132.519572 read 0x536108 thru 0x536848 (0x741 bytes) 000251[0] 1789024132.555328 trunc from 0x9a033e to 0x0f3cb4 (0x8ac689 bytes) 000252[0] 1789024132.622150 read 0x0e89cc thru 0x0e8cbe (0x2f3 bytes) 000253[1] 1789024132.657111 mapread 0x0b078a thru 0x0b106d (0x8e4 bytes) 000255[1] 1789024132.773987 read 0x033cd7 thru 0x0352e0 (0x160a bytes) 000256[0] 1789024132.830345 mapread 0x09b3b6 thru 0x09c732 (0x137d bytes) 000259[0] 1789024132.910091 read 0x058874 thru 0x05a7c8 (0x1f55 bytes) 000260[1] 1789024132.956994 mapread 0x0c5d36 thru 0x0c6ef3 (0x11be bytes) 000261[0] 1789024132.965028 punch from 0x0e657e to 0x0e6fc4 (0xa47 bytes) 000262[1] 1789024133.033491 mapread 0x081414 thru 0x08318a (0x1d77 bytes) 000263[1] 1789024133.078098 read 0x03e31f thru 0x03e399 (0x7b bytes) 000264[1] 1789024133.104431 punch from 0x059504 to 0x05b4fa (0x1ff7 bytes) 000265[0] 1789024133.153822 trunc from 0x0f3cb5 to 0x275459 (0x1817a5 bytes) 000270[1] 1789024133.346281 punch from 0x2223c5 to 0x223289 (0xec5 bytes) 000271[0] 1789024133.380376 punch from 0x15f7c4 to 0x16078d (0xfca bytes) 000273[0] 1789024133.399968 read 0x035f8a thru 0x0363b0 (0x427 bytes) 000276[1] 1789024133.446743 read 0x136859 thru 0x137a47 (0x11ef bytes) 000277[1] 1789024133.552096 mapwrite 0x0a5201 thru 0x0a595b (0x75b bytes) 000280[1] 1789024133.817853 mapwrite 0x6c9652 thru 0x6ca6ac (0x105b bytes) 000284[0] 1789024134.253666 punch from 0x56a0d2 to 0x56a287 (0x1b6 bytes) 000286[1] 1789024134.361424 mapwrite 0x0c64b5 thru 0x0c780d (0x1359 bytes) 000290[1] 1789024134.588669 punch from 0x199a07 to 0x19a8f3 (0xeed bytes) 000291[1] 1789024134.612645 mapwrite 0x0055aa thru 0x00589b (0x2f2 bytes) 000294[0] 1789024134.785913 mapwrite 0x6dca36 thru 0x6dcc5b (0x226 bytes) 000296[1] 1789024135.010381 read 0x362cde thru 0x3631dc (0x4ff bytes) 000297[1] 1789024135.054436 write 0x2c37a6 thru 0x2c4408 (0xc63 bytes) 000298[1] 1789024135.160850 mapread 0x214219 thru 0x214894 (0x67c bytes) 000299[1] 1789024135.232942 read 0x3d5212 thru 0x3d564e (0x43d bytes) 000300[1] 1789024135.392512 read 0x4ec913 thru 0x4ecde9 (0x4d7 bytes) 000301[0] 1789024135.459348 write 0x6c511d thru 0x6c5f77 (0xe5b bytes) 000302[0] 1789024135.512491 read 0x5c1375 thru 0x5c152f (0x1bb bytes) 000303[0] 1789024135.536383 trunc from 0x6dcc5c to 0x2b98a5 (0x4233b6 bytes) 000306[0] 1789024135.631495 trunc from 0x2b98a6 to 0x7b4c68 (0x4fb3c3 bytes) 000308[1] 1789024135.685522 write 0x0f0df4 thru 0x0f1cdd (0xeea bytes) 000310[0] 1789024135.737979 read 0x0fb6d4 thru 0x0fd5d2 (0x1eff bytes) 000312[0] 1789024135.776202 mapwrite 0x361915 thru 0x3623aa (0xa96 bytes) 000314[1] 1789024135.866608 mapread 0x38ac5e thru 0x38b47b (0x81e bytes) 000315[0] 1789024135.901776 trunc from 0x7b4c69 to 0x913d94 (0x15f12c bytes) 000316[1] 1789024135.937105 read 0x7816a4 thru 0x781740 (0x9d bytes) 000317[1] 1789024135.995631 trunc from 0x913d95 to 0x487194 (0x48cc00 bytes) 000319[1] 1789024136.112625 punch from 0x417996 to 0x418185 (0x7f0 bytes) 000322[0] 1789024136.153481 trunc from 0x487195 to 0x634a78 (0x1ad8e4 bytes) 000324[1] 1789024136.225406 write 0x2697ed thru 0x26ab58 (0x136c bytes) 000325[0] 1789024136.398260 trunc from 0x634a79 to 0x9883e7 (0x35396f bytes) 000326[0] 1789024136.502868 write 0x90f2c0 thru 0x910920 (0x1661 bytes) 000327[1] 1789024136.523246 trunc from 0x9883e8 to 0x368c8b (0x61f75c bytes) 000328[1] 1789024136.590067 mapwrite 0x3c196d thru 0x3c2913 (0xfa7 bytes) 000329[0] 1789024136.841872 read 0x161b56 thru 0x162fa0 (0x144b bytes) 000331[0] 1789024136.954496 read 0x293319 thru 0x294cf5 (0x19dd bytes) 000333[1] 1789024137.117432 write 0x6261b4 thru 0x6270cd (0xf1a bytes) 000334[1] 1789024137.125116 read 0x2c47f7 thru 0x2c4ab5 (0x2bf bytes) 000335[1] 1789024137.139438 read 0x4018d1 thru 0x40292f (0x105f bytes) 000337[1] 1789024137.199472 write 0x091641 thru 0x091ca0 (0x660 bytes) 000338[1] 1789024137.296214 mapread 0x38f8e6 thru 0x38fca6 (0x3c1 bytes) 000339[1] 1789024137.380652 mapwrite 0x70ac63 thru 0x70b3a5 (0x743 bytes) 000340[1] 1789024137.570870 trunc from 0x70b3a6 to 0x15ca39 (0x5ae96c bytes) 000341[1] 1789024137.618088 read 0x0e47ef thru 0x0e4a87 (0x299 bytes) 000344[0] 1789024137.703342 read 0x10a7ca thru 0x10b09a (0x8d1 bytes) 000345[0] 1789024137.818749 mapwrite 0x31d03d thru 0x31e55c (0x1520 bytes) 000346[1] 1789024138.039103 punch from 0x25d915 to 0x25ed9e (0x148a bytes) 000347[1] 1789024138.089844 mapwrite 0x054f72 thru 0x055488 (0x517 bytes) 000348[0] 1789024138.247772 punch from 0x1bb502 to 0x1bc4d2 (0xfd1 bytes) 000351[0] 1789024138.325190 write 0x45a489 thru 0x45aa9e (0x616 bytes) 000352[1] 1789024138.351658 mapwrite 0x2dfff9 thru 0x2e0a77 (0xa7f bytes) 000353[0] 1789024138.451973 punch from 0x3a0f4e to 0x3a29ef (0x1aa2 bytes) 000357[1] 1789024138.496003 mapwrite 0x3bbfb7 thru 0x3bd0d7 (0x1121 bytes) 000359[1] 1789024138.658572 punch from 0x239f0e to 0x23a658 (0x74b bytes) 000360[0] 1789024138.679207 mapwrite 0x0ed9f9 thru 0x0ee5e2 (0xbea bytes) 000361[0] 1789024138.787507 trunc from 0x45aa9f to 0x40a39f (0x506ff bytes) 000362[1] 1789024138.840263 read 0x1d6cd6 thru 0x1d7e3c (0x1167 bytes) 000363[0] 1789024138.877329 punch from 0x32c771 to 0x32e5ea (0x1e7a bytes) 000364[0] 1789024138.896399 punch from 0x057677 to 0x057ca4 (0x62e bytes) 000365[0] 1789024138.900265 mapwrite 0x655cf1 thru 0x656a5b (0xd6b bytes) 000366[0] 1789024139.069902 punch from 0x32e606 to 0x32fccc (0x16c7 bytes) 000367[0] 1789024139.083530 trunc from 0x656a5c to 0x096816 (0x5c0245 bytes) 000368[0] 1789024139.112510 trunc from 0x096817 to 0x1b6481 (0x11fc6b bytes) 000369[1] 1789024139.206509 punch from 0x1770cb to 0x17896b (0x18a1 bytes) 000370[0] 1789024139.273403 mapwrite 0x57f3c0 thru 0x57fde4 (0xa25 bytes) 000371[0] 1789024139.425321 read 0x519c9e thru 0x51a2e0 (0x643 bytes) 000375[1] 1789024139.527476 write 0x488401 thru 0x48a06c (0x1c6c bytes) 000376[0] 1789024139.583937 mapread 0x1c14a4 thru 0x1c31ba (0x1d17 bytes) 000378[0] 1789024139.654264 write 0x6389a6 thru 0x63959b (0xbf6 bytes) 000379[0] 1789024139.707638 write 0x685853 thru 0x686a32 (0x11e0 bytes) 000381[1] 1789024139.821910 write 0x330001 thru 0x331788 (0x1788 bytes) 000384[0] 1789024139.987896 trunc from 0x686a33 to 0x8a5158 (0x21e726 bytes) 000386[0] 1789024140.050601 punch from 0x72a2f6 to 0x72a6f7 (0x402 bytes) 000387[0] 1789024140.057238 write 0x9c183f thru 0x9c27e2 (0xfa4 bytes) 000392[1] 1789024140.153065 mapread 0x1bbbb7 thru 0x1bc4ab (0x8f5 bytes) 000395[0] 1789024140.233287 mapwrite 0x34c55b thru 0x34d897 (0x133d bytes) 000397[1] 1789024140.316555 mapwrite 0x6210b4 thru 0x6215d0 (0x51d bytes) 000399[0] 1789024140.490084 write 0x986fe9 thru 0x9875d6 (0x5ee bytes) 000401[1] 1789024140.524764 write 0x4e733f thru 0x4e9335 (0x1ff7 bytes) 000403[0] 1789024140.568451 mapwrite 0x93d0b1 thru 0x93e51b (0x146b bytes) 000405[1] 1789024140.769129 mapwrite 0x8fc1e2 thru 0x8fd416 (0x1235 bytes) 000407[1] 1789024140.950636 mapread 0x602285 thru 0x602fe3 (0xd5f bytes) 000410[0] 1789024141.065347 read 0x2e260d thru 0x2e2937 (0x32b bytes) 000411[1] 1789024141.174508 trunc from 0x9c27e3 to 0x1299b5 (0x898e2d bytes) 000412[0] 1789024141.237210 mapwrite 0x5b8b03 thru 0x5ba959 (0x1e57 bytes) 000416[1] 1789024141.506586 write 0x8650b4 thru 0x865fd2 (0xf1f bytes) 000417[1] 1789024141.534024 mapread 0x073f50 thru 0x075a29 (0x1ada bytes) 000418[0] 1789024141.583481 read 0x865d85 thru 0x865fd2 (0x24e bytes) 000419[1] 1789024141.596611 mapwrite 0x3af634 thru 0x3b1564 (0x1f31 bytes) 000421[0] 1789024141.698441 read 0x019895 thru 0x01ac22 (0x138e bytes) 000422[1] 1789024141.717755 read 0x2f2bcd thru 0x2f4859 (0x1c8d bytes) 000423[0] 1789024141.741562 read 0x50c212 thru 0x50d006 (0xdf5 bytes) 000425[0] 1789024141.747392 mapread 0x1def97 thru 0x1e0a03 (0x1a6d bytes) 000428[0] 1789024141.822362 punch from 0x4825c1 to 0x484311 (0x1d51 bytes) 000431[0] 1789024141.896049 read 0x434b5d thru 0x436564 (0x1a08 bytes) 000432[1] 1789024141.908401 mapread 0x6ce2cf thru 0x6cef69 (0xc9b bytes) 000433[1] 1789024141.953316 read 0x328c75 thru 0x32993a (0xcc6 bytes) 000434[0] 1789024141.968156 read 0x63c48f thru 0x63dd60 (0x18d2 bytes) 000435[1] 1789024141.983097 write 0x4fc4fb thru 0x4fe402 (0x1f08 bytes) 000437[1] 1789024142.038373 punch from 0x80a148 to 0x80bb0c (0x19c5 bytes) 000439[0] 1789024142.141229 trunc from 0x865fd3 to 0x974db4 (0x10ede2 bytes) 000441[0] 1789024142.225396 mapread 0x49bf81 thru 0x49d128 (0x11a8 bytes) 000442[1] 1789024142.277114 mapread 0x9119c6 thru 0x913297 (0x18d2 bytes) 000443[1] 1789024142.349466 trunc from 0x974db5 to 0x041bb3 (0x933201 bytes) 000444[1] 1789024142.413209 trunc from 0x041bb4 to 0x39ef9a (0x35d3e7 bytes) 000446[0] 1789024142.508489 mapwrite 0x140683 thru 0x141ec3 (0x1841 bytes) 000447[1] 1789024142.599175 punch from 0x2e0cd9 to 0x2e259a (0x18c2 bytes) 000449[0] 1789024142.643607 write 0x7ba8e2 thru 0x7baa8a (0x1a9 bytes) 000450[0] 1789024142.665507 mapwrite 0x693197 thru 0x69444d (0x12b7 bytes) 000451[0] 1789024142.742907 write 0x7fc1e3 thru 0x7fd6a9 (0x14c7 bytes) 000452[0] 1789024142.762044 read 0x57ea15 thru 0x580118 (0x1704 bytes) 000453[1] 1789024142.787327 mapwrite 0x560549 thru 0x5623c9 (0x1e81 bytes) 000456[0] 1789024142.917641 write 0x73cdfd thru 0x73e216 (0x141a bytes) 000457[0] 1789024142.964693 read 0x7a8433 thru 0x7a9fad (0x1b7b bytes) 000458[1] 1789024142.990054 trunc from 0x7fd6aa to 0x6b4bf6 (0x148ab3 bytes) 000460[1] 1789024143.113803 trunc from 0x6b4bf7 to 0x23343d (0x4817b9 bytes) 000464[0] 1789024143.188782 write 0x00e469 thru 0x00efed (0xb85 bytes) 000465[0] 1789024143.226126 trunc from 0x23343e to 0x164c4f (0xce7ee bytes) 000471[1] 1789024143.299301 read 0x02f578 thru 0x02fe28 (0x8b1 bytes) 000473[0] 1789024143.406238 mapwrite 0x048fd8 thru 0x04a0fd (0x1126 bytes) 000474[0] 1789024143.495825 punch from 0x016c71 to 0x01848d (0x181d bytes) 000475[1] 1789024143.527236 mapread 0x024644 thru 0x0263a5 (0x1d62 bytes) 000476[1] 1789024143.606842 write 0x026ff2 thru 0x028db3 (0x1dc2 bytes) 000477[1] 1789024143.728400 read 0x141333 thru 0x1420f5 (0xdc3 bytes) 000479[0] 1789024143.901675 trunc from 0x164c50 to 0x894505 (0x72f8b6 bytes) 000480[1] 1789024143.951142 punch from 0x432f90 to 0x434c52 (0x1cc3 bytes) 000482[1] 1789024144.002553 read 0x22c3aa thru 0x22cb57 (0x7ae bytes) 000485[0] 1789024144.113311 mapwrite 0x41d89a thru 0x41f825 (0x1f8c bytes) 000487[0] 1789024144.251438 punch from 0x4f857f to 0x4f9575 (0xff7 bytes) 000490[1] 1789024144.316814 read 0x61fa5e thru 0x62022a (0x7cd bytes) 000491[0] 1789024144.338807 punch from 0x75b89c to 0x75d5aa (0x1d0f bytes) 000492[1] 1789024144.379456 punch from 0x2b6837 to 0x2b7662 (0xe2c bytes) 000493[1] 1789024144.429961 trunc from 0x894506 to 0x0db060 (0x7b94a5 bytes) 000495[1] 1789024144.575934 write 0x4c8205 thru 0x4c8632 (0x42e bytes) 000496[1] 1789024144.582383 punch from 0x244c09 to 0x2465fd (0x19f5 bytes) 000497[0] 1789024144.637472 mapwrite 0x60e6d3 thru 0x60f2b5 (0xbe3 bytes) 000498[1] 1789024144.812066 punch from 0x2de499 to 0x2de6be (0x226 bytes) All operations completed A-OK! PASS 16b (34s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 03:09:10 (1789024150) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 03:09:14 (1789024154) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.398031 s, 21.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.282344 s, 29.7 MB/s ldlm.namespaces.MGC192.168.203.139@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff94be06a1b800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff94be0727e000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff94be0727e000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff94be0727e000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.127 s, 5.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.42777 s, 16.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.53784 s, 29.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.63845 s, 13.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.55322 s, 12.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.58885 s, 29.2 MB/s PASS 16d (62s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 03:10:17 (1789024217) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.264511 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941612 s, 11.1 MB/s PASS 16e (6s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 03:10:23 (1789024223) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 03:10:50 (1789024250) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 03:11:18 (1789024278) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 03:11:25 (1789024285) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 03:11:33 (1789024293) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.41201 s, 18.6 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.42495 s, 18.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.36164 s, 11.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.55039 s, 16.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.1305 s, 12.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.07296 s, 12.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.63124 s, 10.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.79038 s, 14.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.12908 s, 12.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.08375 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.28597 s, 8.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.9192 s, 13.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.09848 s, 12.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.86962 s, 9.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.91981 s, 9.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.32467 s, 11.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.67768 s, 9.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.12101 s, 8.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.12608 s, 8.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (37s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 03:12:10 (1789024330) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Sep 10 03:12:56 EDT 2026 All operations completed A-OK! + fsx 2 done: rc=0 Thu Sep 10 03:12:56 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Thu Sep 10 03:12:57 EDT 2026 + fsx 4 done: rc=0 Thu Sep 10 03:12:57 EDT 2026 + fsx 5 done: rc=0 Thu Sep 10 03:12:57 EDT 2026 PASS 16k (57s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 03:13:07 (1789024387) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 03:13:18 (1789024398) mmap test1: basic mmap operation (PASS, 0.12902s) mmap test2: MAP_PRIVATE not write back (PASS, 0.10582s) mmap test3: concurrent mmap ops on two nodes (PASS, 15.406s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5775s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.16823s) mmap test6: check mmap write/read content on two nodes (PASS, 0.17097s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.1229s) mmap test11: multi-thread shared mmap (PASS, 1.3312s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1093s) PASS 18 (31s) == sanityn test 19: test concurrent uncached read races ========================================================================= 03:13:50 (1789024430) oleg339-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 03:13:54 (1789024434) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 03:14:00 (1789024440) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 03:14:06 (1789024446) now is 1789024448 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 new atime is 1789024509 PASS 23 (68s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 03:15:14 (1789024514) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (5s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 03:15:20 (1789024520) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 03:15:26 (1789024526) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 03:15:33 (1789024533) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 03:15:36 (1789024536) utime: good mknod times 1789024536 <= 1789024537 <= 1789024537 for /mnt/lustre/f26a utime: good mknod times 1789024536 <= 1789024537 <= 1789024537 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 03:15:43 (1789024543) PASS 26b (11s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 03:15:54 (1789024554) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-10 03:15:57.000000000 -0400 Birth: 2026-09-10 03:15:57.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-10 03:15:57.000000000 -0400 Birth: 2026-09-10 03:15:57.000000000 -0400 PASS 26c (10s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 03:16:06 (1789024566) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 03:16:15 (1789024575) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 03:16:23 (1789024583) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg339-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.139@tcp:/lustre 7542784 38912 7499776 1% /mnt/lustre2 PASS 31b (22s) == sanityn test 31r: open-rename(replace) race =========== 03:16:46 (1789024606) -rw-r--r-- 1 root root 0 Sep 10 03:16 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 03:16:55 (1789024615) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43167 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 03:17:02 (1789024622) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 03:17:10 (1789024630) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 03:17:14 (1789024634) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.60988 s, 32.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47283 s, 35.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.95388 s, 26.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.26188 s, 23.2 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.08104 s, 25.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.24185 s, 23.4 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.63806 s, 32.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.66472 s, 31.5 MB/s PASS 32c (30s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 03:17:46 (1789024666) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 03:17:49 (1789024669) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 03:17:52 (1789024672) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 03:17:56 (1789024676) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 03:18:00 (1789024680) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff94be06a1b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94be06a1b800.ost_server_uuid in FULL state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94be06a1b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94be06a1b800.ost_server_uuid in IDLE state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff94be06a1b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94be06a1b800.ost_server_uuid in IDLE state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94be06a1b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94be06a1b800.ost_server_uuid in IDLE state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff94be06a1b800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94be06a1b800.ost_server_uuid in IDLE state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94be06a1b800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94be06a1b800.ost_server_uuid in IDLE state after 0 sec PASS 34 (142s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 03:20:22 (1789024822) fail_loc=0x80000317 Race attempt 0 Wait for 50475 50496 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50475 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.139@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.203.139@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 - open/close 683 (time 1789024839.71 total 10.00 last 68.29) - open/close 1309 (time 1789024849.72 total 20.01 last 62.54) - open/close 1978 (time 1789024859.72 total 30.02 last 66.85) - open/close 2578 (time 1789024869.73 total 40.03 last 59.94) - open/close 3200 (time 1789024879.74 total 50.03 last 62.18) - open/close 3827 (time 1789024889.74 total 60.03 last 62.68) total: 4000 open/close in 64.06 seconds: 62.44 ops/second PASS 35 (74s) == sanityn test 36: handle ESTALE/open-unlink correctly == 03:21:36 (1789024896) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg339-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.10371 s, 47.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7426048) after(7477248) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.679139 s, 77.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7477248) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.03624 s, 50.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7489536) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.678658 s, 77.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7491584) after_dd(7428096) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.923341 s, 56.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7498752) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.803843 s, 65.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7491584) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.714159 s, 73.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(6) *** before(7489536) after_dd(7428096) after(7479296) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08604 s, 48.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7479296) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.8813 s, 59.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.881195 s, 59.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(9) *** before(7491584) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.459461 s, 114 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7488512) after_dd(7438336) after(7498752) PASS 36 (305s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 03:26:42 (1789025202) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 - create 1029 (time 1789025217.41 total 10.02 last 102.74) - create 2152 (time 1789025227.41 total 20.02 last 112.27) - create 3371 (time 1789025237.42 total 30.02 last 121.83) - create 4622 (time 1789025247.42 total 40.03 last 125.09) - create 5754 (time 1789025257.42 total 50.03 last 113.14) - create 6947 (time 1789025267.43 total 60.03 last 119.26) - create 8169 (time 1789025277.43 total 70.04 last 122.12) - create 9361 (time 1789025287.44 total 80.05 last 119.13) total: 10000 create in 85.74 seconds: 116.64 ops/second PASS 37 (98s) == sanityn test 39a: file mtime does not change after rename ========================================================== 03:28:20 (1789025300) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 03:28:27 (1789025307) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 03:28:36 (1789025316) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 03:28:45 (1789025325) fail_loc=0x411 fail_loc=0 PASS 39d (9s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 03:28:56 (1789025336) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (19s) == sanityn test 40c: pdirops: link and others ======================================================================== 03:29:15 (1789025355) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (23s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 03:29:39 (1789025379) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (24s) == sanityn test 40e: pdirops: rename and others ======================================================================== 03:30:02 (1789025402) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (19s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 03:30:21 (1789025421) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 03:30:35 (1789025435) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 03:30:49 (1789025449) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (15s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 03:31:03 (1789025463) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 03:31:16 (1789025476) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 03:31:29 (1789025489) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 03:31:44 (1789025504) Conflict PASS 41g (17s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 03:32:02 (1789025522) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (16s) == sanityn test 41i: reint_open: create vs create ======== 03:32:19 (1789025539) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1109s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 03:50:47 (1789026647) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (15s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 03:51:02 (1789026662) open: File exists Conflict PASS 42b (14s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 03:51:16 (1789026676) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 03:51:31 (1789026691) Conflict PASS 42d (19s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 03:51:50 (1789026710) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (18s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 03:52:08 (1789026728) Conflict PASS 42f (15s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 03:52:23 (1789026743) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 03:52:36 (1789026756) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 03:52:51 (1789026771) PASS 43a (75s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 03:54:06 (1789026846) Conflict PASS 43b (13s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 03:54:19 (1789026859) Conflict PASS 43c (13s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 03:54:32 (1789026872) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (12s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 03:54:44 (1789026884) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 03:54:57 (1789026897) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 03:55:11 (1789026911) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 03:55:24 (1789026924) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 03:55:38 (1789026938) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 03:55:42 (1789026942) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (117s) == sanityn test 43k: unlink vs create ==================== 03:57:39 (1789027059) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1155s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 04:16:54 (1789028214) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Sep 10 04:17:02 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 04:17:08 (1789028228) open: File exists Conflict PASS 44b (18s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 04:17:26 (1789028246) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 04:17:40 (1789028260) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 04:17:54 (1789028274) Conflict PASS 44e (16s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 04:18:10 (1789028290) Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 04:18:25 (1789028305) Conflict PASS 44g (13s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 04:18:38 (1789028318) Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 04:18:52 (1789028332) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 04:18:55 (1789028335) PASS 45a (124s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 04:21:00 (1789028460) Conflict PASS 45b (14s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 04:21:14 (1789028474) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 04:21:28 (1789028488) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (16s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 04:21:44 (1789028504) Conflict PASS 45e (17s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 04:22:01 (1789028521) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (17s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 04:22:18 (1789028538) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (17s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 04:22:35 (1789028555) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 04:22:49 (1789028569) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 04:22:53 (1789028573) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1314s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 04:44:47 (1789029887) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (16s) == sanityn test 46b: pdirops: link vs create ======================================================================== 04:45:04 (1789029904) open: File exists Conflict PASS 46b (19s) == sanityn test 46c: pdirops: link vs link ======================================================================== 04:45:22 (1789029922) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (19s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 04:45:41 (1789029941) Conflict PASS 46d (16s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 04:45:57 (1789029957) Conflict PASS 46e (16s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 04:46:13 (1789029973) Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 04:46:28 (1789029988) Conflict PASS 46g (15s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 04:46:44 (1789030004) Conflict PASS 46h (17s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 04:47:01 (1789030021) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (6s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 04:47:07 (1789030027) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (5s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 04:47:12 (1789030032) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (7s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 04:47:19 (1789030039) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (5s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 04:47:24 (1789030044) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 04:47:29 (1789030049) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 04:47:33 (1789030053) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 04:47:37 (1789030057) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 04:47:40 (1789030060) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.351578 s, 29.1 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 04:47:53 (1789030073) 0+1 records in 0+1 records out 158 bytes copied, 0.0550803 s, 2.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218483) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 04:48:03 (1789030083) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0242049 s, 42.3 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825869 s, 12.7 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 04:48:24 (1789030104) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0321835 s, 31.8 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 04:48:37 (1789030117) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573341 s, 18.3 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 04:48:46 (1789030126) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 04:48:56 (1789030136) llite.lustre-ffff94be06a1b800.xattr_cache=1 llite.lustre-ffff94be0727e000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 04:48:58.000000000 -0400 Modify: 2026-09-10 04:48:58.000000000 -0400 Change: 2026-09-10 04:48:58.000000000 -0400 Birth: 2026-09-10 04:48:58.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 04:49:05.000000000 -0400 Modify: 2026-09-10 04:49:06.000000000 -0400 Change: 2026-09-10 04:49:06.000000000 -0400 Birth: 2026-09-10 04:49:05.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 04:49:11.000000000 -0400 Modify: 2026-09-10 04:49:11.000000000 -0400 Change: 2026-09-10 04:49:11.000000000 -0400 Birth: 2026-09-10 04:49:12.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 04:49:11.000000000 -0400 Modify: 2026-09-10 04:49:12.000000000 -0400 Change: 2026-09-10 04:49:12.000000000 -0400 Birth: 2026-09-10 04:49:12.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 04:49:17.000000000 -0400 Modify: 2026-09-10 04:49:19.000000000 -0400 Change: 2026-09-10 04:49:19.000000000 -0400 Birth: 2026-09-10 04:49:18.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 04:49:17.000000000 -0400 Modify: 2026-09-10 04:49:19.000000000 -0400 Change: 2026-09-10 04:49:19.000000000 -0400 Birth: 2026-09-10 04:49:18.000000000 -0400 PASS 54 (33s) == sanityn test 55a: rename vs unlink target dir ========= 04:49:29 (1789030169) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (15s) == sanityn test 55b: rename vs unlink source dir ========= 04:49:44 (1789030184) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (15s) == sanityn test 55c: rename vs unlink orphan target dir == 04:50:00 (1789030200) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (21s) == sanityn test 55d: rename file vs link ================= 04:50:21 (1789030221) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (17s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 04:50:39 (1789030239) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 04:50:43 (1789030243) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 04:51:04 (1789030264) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (27s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 04:51:31 (1789030291) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789030294 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.544992 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (122s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 04:53:34 (1789030414) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789030418 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.06151 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (129s) == sanityn test 60: Verify data_version behaviour ======== 04:55:42 (1789030542) PASS 60 (9s) == sanityn test 70a: cd directory && rm directory ======== 04:55:51 (1789030551) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 04:56:00 (1789030560) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 04:56:10 (1789030570) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00887813 s, 7.4 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 04:56:14 (1789030574) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0224981 s, 2.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (5s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 04:56:18 (1789030578) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 04:56:22 (1789030582) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0273266 s, 2.4 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 04:56:26 (1789030586) llite.lustre-ffff94be06a1b800.xattr_cache=1 llite.lustre-ffff94be0727e000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (9s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 04:56:34 (1789030594) llite.lustre-ffff94be06a1b800.xattr_cache=1 llite.lustre-ffff94be0727e000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 04:56:43 (1789030603) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0342549 s, 29.9 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.040182 s, 25.5 kB/s 231303: get lock1 231301: get lock2 231303: done 231303: get lock2 231301: done 231301: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231301: cannot get lock1: Resource deadlock avoided 231303: done 231303: exit rc=0 231301: done 231301: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 04:56:54 (1789030614) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0401065 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300036 s, 34.9 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231884 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 04:57:06 (1789030626) opening files: ........ closing files: ........ PASS 76 (58s) == sanityn test 77a: check FIFO NRS policy =============== 04:58:05 (1789030685) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.23825 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550594 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509447 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657387 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12898 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134733 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189653 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161477 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166767 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203548 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182577 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185124 s, 5.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238266 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251284 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230018 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298677 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22058 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112974 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10609 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012515 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432005 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418848 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11847 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520206 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141305 s, 74.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857562 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067027 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120082 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653084 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340083 s, 30.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147908 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727461 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366515 s, 28.6 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (10s) == sanityn test 77b: check CRR-N NRS policy ============== 04:58:15 (1789030695) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.337472 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345866 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570093 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661444 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118576 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12555 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148869 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141378 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181861 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205269 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171241 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203022 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234886 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223179 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211765 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273254 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29978 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740996 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105486 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878869 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669356 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624075 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590454 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130776 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116752 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713087 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151693 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145736 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142402 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092566 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102726 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898867 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118537 s, 8.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.288315 s, 58.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042895 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378309 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10215 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897098 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585659 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829275 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283039 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11066 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513935 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370298 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199516 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719863 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617079 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635295 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649405 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136479 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823534 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571448 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985746 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058017 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635043 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680178 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110476 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083499 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886775 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957026 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103747 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103807 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119855 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109595 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119314 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132998 s, 7.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (14s) == sanityn test 77c: check ORR NRS policy ================ 04:58:29 (1789030709) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.354031 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312098 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673024 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129145 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126832 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879538 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211295 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449959 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141882 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569271 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126267 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220064 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272527 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157742 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244058 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242153 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194313 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104988 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767287 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600637 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00909611 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832902 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00746902 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00700982 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109623 s, 95.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014236 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192852 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133371 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610552 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107845 s, 97.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527667 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583888 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00907046 s, 116 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.31758 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510419 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175464 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810534 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432023 s, 24.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211124 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856956 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208975 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196495 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172209 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16605 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149475 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236566 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252651 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269994 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173451 s, 6.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238452 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305041 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576463 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362499 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127904 s, 82.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694619 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282153 s, 37.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326489 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691933 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651963 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470462 s, 22.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778045 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00972223 s, 108 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490986 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127732 s, 82.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850725 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00492135 s, 213 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.273457 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332941 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801587 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143183 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073107 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118948 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108006 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442724 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175259 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139192 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1978 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820155 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21429 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228047 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813508 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191093 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220366 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552076 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060446 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10456 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214879 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720467 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574439 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921262 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134135 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219956 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00280548 s, 374 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105779 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246097 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998198 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127182 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989973 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410464 s, 25.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 04:58:46 (1789030726) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.316689 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241729 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282113 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336086 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709037 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457265 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155932 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729164 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856521 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958001 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453892 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490807 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138774 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18043 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329488 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232744 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246043 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153239 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125681 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940767 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158016 s, 66.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167638 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169879 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888891 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748305 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100915 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17751 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105361 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130122 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353473 s, 29.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04276 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277036 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358422 s, 29.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.253706 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470772 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832118 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871484 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137346 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139908 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110742 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180472 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135092 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217162 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139059 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222491 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199492 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244109 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248741 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22285 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201097 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00483875 s, 217 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00279729 s, 375 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360202 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00279813 s, 375 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00287039 s, 365 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637297 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885265 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143961 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064776 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798102 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111902 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485927 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273806 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605693 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474106 s, 22.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.40188 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642844 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781631 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821786 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15024 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15826 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137339 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123333 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203966 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183832 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232659 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210911 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179369 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270086 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204828 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266972 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309826 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927298 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167248 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106549 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711654 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191495 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655028 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104653 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345515 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00590271 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386949 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10446 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448148 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471482 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562219 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13495 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922749 s, 11.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (17s) == sanityn test 77e: check TBF NID nrs policy ============ 04:59:03 (1789030743) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.39@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.271359 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684702 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638909 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174917 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768618 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103641 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878386 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134797 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233758 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240022 s, 4.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205434 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19549 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248944 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271204 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245236 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297509 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319787 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14386 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18617 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208947 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277487 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143444 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12488 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130102 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756794 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157576 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150354 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144286 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144983 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28173 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304702 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626685 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778297 s, 13.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.638532 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652385 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12769 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117131 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145217 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106136 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162921 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200639 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157956 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200675 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237478 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182545 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262425 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174498 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256923 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27135 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326832 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709075 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109336 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161214 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114443 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14072 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113837 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693083 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140975 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16477 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125864 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138936 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167389 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124588 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133033 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209878 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154475 s, 6.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.407058 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490062 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067447 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127659 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089714 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323146 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683761 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145341 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126356 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682078 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175804 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622948 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121329 s, 8.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107263 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146372 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179779 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327265 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148303 s, 70.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012167 s, 86.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154024 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895597 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652453 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542618 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493307 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184634 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00277585 s, 378 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221531 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129916 s, 80.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225487 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00968841 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166193 s, 63.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233339 s, 44.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00956429 s, 110 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224425 s, 74.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675218 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624291 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723176 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09876 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978942 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20952 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319483 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294842 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316779 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206289 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31023 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302542 s, 3.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273342 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316447 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378343 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379207 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100121 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00804508 s, 130 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022574 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00268414 s, 391 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170256 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112527 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131779 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170465 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109256 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114139 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780752 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832895 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697425 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689678 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518202 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403019 s, 26.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (28s) == sanityn test 77f: check TBF JobID nrs policy ========== 04:59:31 (1789030771) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25596 s, 65.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055838 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630304 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475032 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400909 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294444 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017971 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143727 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879262 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854832 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21111 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12532 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234781 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133937 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23697 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127612 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226573 s, 4.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622698 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961964 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105675 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11465 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401244 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083312 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084292 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436816 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798417 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948144 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695148 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829677 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111643 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931066 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838634 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088821 s, 11.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.382685 s, 43.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202815 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445438 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283138 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549591 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483624 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033672 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858988 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817968 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103522 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182959 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121992 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167531 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17483 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173597 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171573 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191866 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675892 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056693 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556304 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371307 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038482 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385638 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046628 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958768 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699342 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628725 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049387 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070156 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690137 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732417 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826259 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926018 s, 11.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.266036 s, 63.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204526 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308368 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537099 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897838 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960838 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106194 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366497 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759032 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108878 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126956 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116264 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129504 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148943 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757525 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136855 s, 7.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158992 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833456 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815767 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713812 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635916 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502985 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704305 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110534 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103212 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11723 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090928 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10127 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104706 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10468 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117387 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132222 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14228 s, 7.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.302841 s, 55.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248048 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315249 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595041 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608461 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394348 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538764 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920915 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699485 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586843 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591908 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589228 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638924 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047042 s, 22.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774161 s, 13.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131313 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141254 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00611215 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00961521 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525184 s, 20.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509501 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354476 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388867 s, 27.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297725 s, 35.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667103 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393356 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275528 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526786 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00770124 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367691 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509755 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496441 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437261 s, 24.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (22s) == sanityn test 77g: Change TBF type directly ============ 04:59:53 (1789030793) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (13s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 05:00:06 (1789030806) oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 PASS 77h (11s) == sanityn test 77i: Change rank of TBF rule ============= 05:00:17 (1789030817) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (23s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 05:00:40 (1789030840) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.485134 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639449 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804814 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666305 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822898 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729439 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212869 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116573 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155536 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232685 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212848 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124096 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143784 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168377 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181664 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250244 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176782 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759877 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734261 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575776 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618894 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835892 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793909 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623429 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832905 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913919 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870857 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774079 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792613 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108189 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106822 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874653 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142888 s, 7.3 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.29733 s, 11.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6251 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (69s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 05:01:49 (1789030909) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30118 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580096 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829624 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914693 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540343 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175528 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161321 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181496 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204096 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218826 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185237 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1738 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176959 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161392 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178739 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269137 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317103 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047838 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745382 s, 14.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672992 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754589 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553418 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564206 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079064 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539617 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619775 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664338 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544788 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727869 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100871 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611481 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674569 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726889 s, 14.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9732 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9701 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30062 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685803 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915403 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100479 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782733 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119152 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955222 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159773 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13341 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200496 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182377 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158634 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197247 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207846 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201244 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162441 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176878 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879422 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844043 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107594 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108145 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111101 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117938 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099378 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117098 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134743 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134763 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106289 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115892 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145177 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147586 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134069 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168468 s, 6.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8698 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9323 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (154s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 05:04:23 (1789031063) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30181 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431941 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969123 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822918 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869271 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154084 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155911 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178419 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195404 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153813 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15862 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244593 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113172 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147884 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239184 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145662 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192249 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103397 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102816 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790144 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876513 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124364 s, 8.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619313 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13391 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119773 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670967 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679399 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846456 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102052 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124087 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117335 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115573 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1264 s, 8.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9144 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9749 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30375 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750442 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981502 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119774 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143741 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176835 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135387 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285652 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223539 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226169 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259743 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239828 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203777 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320011 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196944 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273552 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264165 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586435 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172351 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859824 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883117 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871062 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708043 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480808 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117539 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712168 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628003 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113633 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619024 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116661 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065668 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690837 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782269 s, 13.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.888 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9244 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (154s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 05:06:57 (1789031217) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.587515 s, 28.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736387 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522252 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513598 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100401 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572366 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619246 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746547 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167658 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185719 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205104 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20078 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118572 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242542 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023983 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623055 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253095 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be06a1b800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00485363 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077321 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209066 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831286 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476599 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388643 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0070186 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00220081 s, 476 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213789 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111738 s, 93.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144783 s, 72.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00355385 s, 295 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237407 s, 442 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00838915 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00199377 s, 526 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129058 s, 81.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.18993 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95679 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.39@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.39@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.522159 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264482 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336611 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637149 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753504 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111837 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190093 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13056 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114121 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127778 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131324 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204401 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194931 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199606 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221719 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213126 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322533 s, 3.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff94be06a1b800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517166 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104418 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401396 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042011 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371323 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547174 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576999 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642526 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696284 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538634 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730919 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856908 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937264 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769956 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101823 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102696 s, 10.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80783 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94754 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.39@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.43685 s, 38.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348533 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483689 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405072 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933614 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870345 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951883 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786739 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678762 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115087 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114693 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571893 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137915 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077512 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740112 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896935 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18325 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941173 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111846 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962731 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908351 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938825 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926259 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105871 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112261 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133126 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123902 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978216 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103637 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105136 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106242 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127766 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127983 s, 8.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.53965 s, 11.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.22951 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.526041 s, 31.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644573 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541573 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101096 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543482 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125299 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155565 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120804 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238001 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262442 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250567 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248679 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296271 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316163 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328888 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358446 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37608 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023089 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379638 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325999 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041357 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460656 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562628 s, 18.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624228 s, 16.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775894 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688064 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13866 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767007 s, 13.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746133 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778835 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778009 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837204 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910524 s, 11.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1865 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93511 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.11744 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96302 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.929503 s, 18.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326921 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497269 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103666 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159352 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15285 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133806 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116254 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138341 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128673 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130286 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17746 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147775 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155431 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166756 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159787 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220202 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677489 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037985 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464611 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884812 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504353 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572466 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045455 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654258 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07226 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604726 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747194 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701107 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627422 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828346 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752895 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895201 s, 11.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.849 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6005 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.400947 s, 41.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475175 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395022 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110313 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1073 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916955 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100324 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126271 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139536 s, 7.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147336 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113592 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142982 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171807 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142366 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1311 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168746 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208053 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720895 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753096 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671282 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777638 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655884 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655498 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821909 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813506 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671063 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770606 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945905 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752142 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828414 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101301 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903019 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106424 s, 9.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.09662 s, 14.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8698 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.66358 s, 25.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395692 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484829 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635789 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584371 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905587 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10359 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155378 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982479 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138675 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14491 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170112 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154232 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110388 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167188 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117542 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120122 s, 8.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460402 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791075 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452376 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521718 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855811 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500733 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450205 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613483 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602359 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694913 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711466 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671785 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737231 s, 14.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10659 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689478 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777476 s, 13.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3629 s, 10.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86033 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2355 s, 10.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88247 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (431s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 05:14:09 (1789031649) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.527339 s, 31.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421406 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042364 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115282 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133874 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164661 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141476 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153275 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126565 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164221 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265687 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190579 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281165 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242969 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280218 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232501 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307589 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369602 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492663 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491646 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819901 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607753 s, 17.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903771 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636335 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455643 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732104 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651251 s, 16.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423753 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454097 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750499 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482276 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516352 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084428 s, 12.4 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.82589 s, 15.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92647 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (47s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 05:14:56 (1789031696) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.39@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.39@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.429343 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427686 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527482 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379855 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221333 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491476 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127314 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811704 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1018 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152236 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109939 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693411 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155388 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742053 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130398 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105342 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108373 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419481 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358866 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554225 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492608 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044785 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629115 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507079 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571613 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112509 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585727 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047491 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466589 s, 22.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048653 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558162 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546127 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568581 s, 18.4 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.22195 s, 14.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91692 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (48s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 05:15:44 (1789031744) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.39@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.367728 s, 45.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430568 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435519 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101556 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939774 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119124 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150923 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990189 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137016 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133483 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163023 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159624 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112529 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220959 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207002 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222434 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268841 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693416 s, 15.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876863 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485111 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367826 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279007 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044487 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103682 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110587 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05905 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525048 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453441 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832014 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586513 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765913 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781873 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962918 s, 10.9 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.74164 s, 13.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12347 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (46s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 05:16:30 (1789031790) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.60726 s, 27.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160429 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563683 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07714 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862308 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114718 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913238 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153225 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140449 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15828 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147534 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113363 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140916 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209056 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153519 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211606 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167619 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024311 s, 431 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128634 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945669 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12625 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116498 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870292 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114626 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155089 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178965 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133906 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602987 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909039 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681308 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051702 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588651 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055352 s, 18.9 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7713 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92608 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.0102 s, 15.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94492 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (91s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 05:18:01 (1789031881) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.04285 s, 16.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478205 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075804 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504085 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03299 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815802 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108555 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178968 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144498 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186004 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953793 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188175 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141142 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119754 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172891 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141004 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252053 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792399 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671285 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424988 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065355 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932006 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477261 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497716 s, 21.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511057 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720868 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560705 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058289 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820154 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679651 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676973 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956963 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782133 s, 13.4 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7895 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.95 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (72s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 05:19:13 (1789031953) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.316702 s, 53.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236311 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041853 s, 25.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107092 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926912 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707385 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914956 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104941 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103316 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989709 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123696 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156764 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142589 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117755 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162784 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140548 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168252 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596968 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665358 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023126 s, 45.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262639 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356442 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314502 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337337 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440065 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444925 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468429 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516083 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504802 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534747 s, 19.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535195 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544337 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801866 s, 13.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.54509 s, 13.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90317 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.523232 s, 32.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300366 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524088 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567163 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354319 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490502 s, 21.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10886 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875838 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124319 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162191 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134551 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926506 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567433 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158519 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204839 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25095 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292464 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257135 s, 40.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462815 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395436 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715514 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373284 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554623 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400005 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515846 s, 20.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445641 s, 23.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112808 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123085 s, 8.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779816 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06318 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701724 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763402 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09964 s, 10.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3247 s, 10.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91908 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4168 s, 10.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93108 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (134s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 05:21:27 (1789032087) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:) Stopping client oleg339-client.virtnet /mnt/lustre opts: Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg339-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg339-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.39,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg339-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.39,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg339-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.39,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69982/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.39,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg339-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: kmemleak disabled oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Starting client oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Started clients oleg339-client.virtnet: 192.168.203.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Starting client oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Started clients oleg339-client.virtnet: 192.168.203.139@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff94be07047000.idle_timeout=debug osc.lustre-OST0000-osc-ffff94be18ba6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff94be07047000.idle_timeout=debug osc.lustre-OST0001-osc-ffff94be18ba6000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.465444 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841872 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713039 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899448 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101543 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25174 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216987 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257907 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242411 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249362 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278806 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302405 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250354 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306614 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314192 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331636 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288718 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be07047000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732268 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634112 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370908 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435303 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361751 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300445 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480375 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053279 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526426 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549066 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803853 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427111 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537728 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489241 s, 21.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562843 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625745 s, 16.8 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.13011 s, 12.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.16 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.605877 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540022 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519522 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592417 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685228 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056772 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145005 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130571 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174151 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177758 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166895 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172827 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228233 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248538 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228677 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270184 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217714 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be07047000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff94be07047000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185906 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188883 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191704 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181477 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200736 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205954 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210462 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225967 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219994 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227055 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226028 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243895 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238837 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234955 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be07047000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272022 s, 3.9 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.38512 s, 14.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0513 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (201s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 05:24:48 (1789032288) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (15s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 05:25:03 (1789032303) On MGS 192.168.203.139, active = nodemap.active=1 On MGS 192.168.203.139, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.55686 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718273 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936774 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914009 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948126 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164651 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178255 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192943 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258564 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224908 s, 4.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262754 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be07047000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff94be07047000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285378 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29767 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310928 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268859 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260044 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318661 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425772 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421537 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422687 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425985 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428963 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431106 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434777 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438279 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453306 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450952 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452621 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454385 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.510992 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46476 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.469515 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.478706 s, 2.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.58812 s, 15.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06723 s, 20.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.139, TBF.id = On MGS 192.168.203.139, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.486491 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729305 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905659 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755618 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108652 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819324 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802798 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865218 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136112 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104348 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586679 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201387 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00703265 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560182 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693864 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207753 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265755 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711757 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027235 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787164 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230427 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025323 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00352417 s, 298 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297577 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313825 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be07047000.lock_unused_count=9 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194082 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628154 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250955 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282025 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230011 s, 45.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129835 s, 80.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118376 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00627051 s, 167 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.43363 s, 14.1 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.06188 s, 34.2 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 On MGS 192.168.203.139, active = nodemap.active=1 On MGS 192.168.203.139, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.449643 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031806 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048623 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591887 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144236 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208686 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20292 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263016 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241801 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3222 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352755 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333652 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365683 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385437 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391382 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380227 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392806 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be07047000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff94be07047000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108266 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155784 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181529 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194994 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162343 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19693 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118347 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209285 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12699 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180396 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216942 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13566 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134972 s, 7.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134781 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14097 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232465 s, 4.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.98685 s, 15.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85628 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.139, TBF.id = On MGS 192.168.203.139, active = nodemap.active=0 PASS 77kj (161s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 05:27:44 (1789032464) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 05:27:55 (1789032475) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg339-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.65592 s, 287 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg339-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0568 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 05:28:53 (1789032533) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.402632 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650016 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981908 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699795 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500821 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125428 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114656 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262222 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285811 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346891 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27612 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263153 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366237 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385674 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440064 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394958 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff94be07047000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff94be07047000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749504 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203828 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137589 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619278 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568921 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488058 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724041 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744683 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498681 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115035 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118784 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732051 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907401 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093231 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131761 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942336 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838843 s, 12.5 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.7723 s, 15.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06394 s, 20.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.392998 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451497 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670814 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984626 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177537 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160135 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20066 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197423 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172297 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24195 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265085 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227739 s, 4.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255364 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190805 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187951 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221474 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247849 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469506 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654389 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503433 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539565 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540016 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447892 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336633 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639376 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528573 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682157 s, 15.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04173 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454673 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444863 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607143 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486149 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504651 s, 20.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.33931 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09049 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (82s) == sanityn test 77o: Changing rank should not panic ====== 05:30:15 (1789032615) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (11s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 05:30:26 (1789032626) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (100s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 05:32:06 (1789032726) mds.MDS.mdt.nrs_policies=tbf oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg339-client: oleg339-server: ssh exited with exit code 36 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (34s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 05:32:40 (1789032760) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273270 kill md_thread xargs PASS 77r (49s) == sanityn test 77s: Check TBF LRU shrinker ============== 05:33:30 (1789032810) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (64s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 05:34:33 (1789032873) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (7s) == sanityn test 79: xattr: intent error ================== 05:34:40 (1789032880) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935607 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 05:34:42.000000000 -0400 Modify: 2026-09-10 05:34:42.000000000 -0400 Change: 2026-09-10 05:34:42.000000000 -0400 Birth: 2026-09-10 05:34:42.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (17s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 05:34:58 (1789032898) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (2s) == sanityn test 80b: Accessing directory during migration ========================================================== 05:35:01 (1789032901) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 05:35:04 (1789032904) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 05:35:07 (1789032907) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 05:35:11 (1789032911) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 05:35:14 (1789032914) - open/close 685 (time 1789032930.11 total 10.01 last 68.44) - open/close 1328 (time 1789032940.11 total 20.01 last 64.29) - open/close 2026 (time 1789032950.12 total 30.02 last 69.76) total: 2500 open/close in 36.23 seconds: 69.00 ops/second snapshot_time 1789033051.476536112 secs.nsecs start_time 1789032918.193139170 secs.nsecs elapsed_time 133.283396942 secs.nsecs open 2505 samples [usecs] 127 15752 3770062 8929172464 close 2504 samples [usecs] 136 38715 962030 2210567844 mknod 2501 samples [usecs] 570 10785 3270436 6798263338 unlink 2500 samples [usecs] 743 81226 5679620 50722133540 mkdir 2 samples [usecs] 1648 4715 6363 24947129 rmdir 2 samples [usecs] 7329 10720 18049 168632641 rename 2500 samples [usecs] 476 53978 18045024 250790753936 getattr 3761 samples [usecs] 10 12846 294443 524510971 parallel_rename_file 2500 samples [usecs] 476 53978 18045024 250790753936 crossdir_rename 2500 samples [usecs] 476 53978 18045024 250790753936 readpage 21 samples [usecs] 5341 137049 523162 28680749930 PASS 81d (142s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 05:37:37 (1789033057) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7614 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7614 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 05:37:43 (1789033063) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 05:37:47 (1789033067) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 05:38:00 (1789033080) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789033082 Finishing test test1 at 1789033086 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 05:38:12 (1789033092) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 05:38:15 (1789033095) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 05:38:18 (1789033098) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 05:38:21 (1789033101) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5766 0x1686 0x240000400 1 5695 0x163f 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5767 0x1687 0x240000400 1 5696 0x1640 0x280000400 rc1=0 and rc2=0 PASS 93 (15s) == sanityn test 94: signal vs CP callback race =========== 05:38:36 (1789033116) osc.lustre-OST0000-osc-ffff94be07047000.idle_timeout=0 osc.lustre-OST0000-osc-ffff94be18ba6000.idle_timeout=0 osc.lustre-OST0001-osc-ffff94be07047000.idle_timeout=0 osc.lustre-OST0001-osc-ffff94be18ba6000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.217957 s, 37.6 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.99011 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 289729 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0463028 s, 88.5 kB/s osc.lustre-OST0000-osc-ffff94be07047000.idle_timeout=debug osc.lustre-OST0000-osc-ffff94be18ba6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff94be07047000.idle_timeout=debug osc.lustre-OST0001-osc-ffff94be18ba6000.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 05:38:51 (1789033131) llite.lustre-ffff94be07047000.fast_read=0 llite.lustre-ffff94be18ba6000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0537139 s, 39.0 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff94be07047000.fast_read=1 llite.lustre-ffff94be18ba6000.fast_read=1 PASS 95a (19s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 05:39:10 (1789033150) llite.lustre-ffff94be07047000.fast_read=0 llite.lustre-ffff94be18ba6000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0100715 s, 1.2 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff94be07047000.fast_read=1 llite.lustre-ffff94be18ba6000.fast_read=1 PASS 95b (12s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 05:39:22 (1789033162) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 05:39:26 (1789033166) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00639255 s, 641 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 05:39:33 (1789033173) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.143258 s, 14.6 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (6s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 05:39:39 (1789033179) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145466 s, 14.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 05:39:46 (1789033186) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00897997 s, 1.3 MB/s PASS 100e (6s) == sanityn test 101a: Discard DoM data on unlink ========= 05:39:52 (1789033192) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0183081 s, 224 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 05:39:59 (1789033199) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0148981 s, 275 kB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 05:40:06 (1789033206) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0110584 s, 370 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 05:40:14 (1789033214) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 18 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf18:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (7s) == sanityn test 103: Test size correctness with lockahead ========================================================== 05:40:21 (1789033221) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789033223 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789033227 Finishing test test23 at 1789033227 PASS 103 (12s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 05:40:33 (1789033233) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 05:40:36 (1789033236) fail_loc=0x416 PASS 105 (24s) == sanityn test 106a: Verify the btime via statx() ======= 05:41:00 (1789033260) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 05:41:05 (1789033265) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0109254 s, 93.7 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938143 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 05:41:06.000000000 -0400 Modify: 2026-09-10 05:41:07.000000000 -0400 Change: 2026-09-10 05:41:07.000000000 -0400 Birth: 2026-09-10 05:41:07.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938143 -rw-r--r-- 1 0 0 1789033267 1789033266 1789033267 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938143 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 05:41:06.000000000 -0400 Modify: 2026-09-10 05:41:07.000000000 -0400 Change: 2026-09-10 05:41:07.000000000 -0400 Birth: 2026-09-10 05:41:07.000000000 -0400 1789033267 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 05:41:13 (1789033273) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 05:41:21 (1789033281) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.391565 s, 26.8 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5776 0x1690 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 05:41:30 (1789033290) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.254726 s, 41.2 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5703 0x1647 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 05:41:43 (1789033303) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0113258 s, 883 kB/s llite.lustre-ffff94be07047000.hybrid_io=0 llite.lustre-ffff94be18ba6000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.57684 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 05:41:55 (1789033315) 192.168.203.139@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre 192.168.203.139@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: kmemleak disabled oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 PASS 109 (45s) == sanityn test 110: do not grant another lock on resend ========================================================== 05:42:40 (1789033360) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 05:42:42.000000000 -0400 Modify: 2026-09-10 05:42:42.000000000 -0400 Change: 2026-09-10 05:42:42.000000000 -0400 Birth: 2026-09-10 05:42:43.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg339-client.virtnet oleg339-client.virtnet PASS 110 (83s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 05:44:03 (1789033443) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 05:44:07 (1789033447) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 05:44:10 (1789033450) lustre-OST0000-osc-ffff94be0543e000 active. lustre-OST0001-osc-ffff94be0543e000 active. lustre-OST0000-osc-ffff94be0543e000 active. lustre-OST0001-osc-ffff94be0543e000 active. MGC192.168.203.139@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 05:44:17 (1789033457) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 05:44:20 (1789033460) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 05:44:23 (1789033463) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 05:44:27 (1789033467) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0137188 s, 1.5 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.110688 s, 185 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 05:44:36 (1789033476) PASS 122 (10s) == sanityn test 200: service remains healthy while able to process request ========================================================== 05:44:46 (1789033486) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=8 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2136 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1431 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.274 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2023 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2462 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2213 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1547 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2426 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2474 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2131 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2346 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3189 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3588 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3812 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3817 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.35 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff94be05238000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94be05238000.ost_server_uuid in IDLE state after 0 sec PASS 200 (82s) cleanup: ====================================================== == sanityn test complete, duration 9937 sec ============== 05:46:09 (1789033569) === sanityn: start cleanup 05:46:11 (1789033571) === Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg339-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 05:50:52 (1789033852) ===