-----============= acceptance-small: sanityn ============----- Sun Sep 6 21:27:46 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg147-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 21:28:07 (1788744487) === Starting client oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 Mount client oleg147-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 Started clients oleg147-client.virtnet: 192.168.201.147@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg147-client.virtnet: executing check_config_client /mnt/lustre oleg147-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg147-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9aa4c2ebe000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9aa4c8847000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa4c2ebe000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa4c8847000.idle_timeout=debug disable quota as required oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 21:28:45 (1788744525) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503648 s, 20.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7646] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 21:28:48 (1788744528) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 21:28:58 (1788744538) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 21:29:06 (1788744546) -rw-r--r-- 1 root root 0 Sep 6 21:29 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (4s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 21:29:11 (1788744551) -rw-r--r-- 1 root root 0 Sep 6 21:29 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (5s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 21:29:16 (1788744556) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 21:29:22 (1788744562) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 21:29:28 (1788744568) total 1 -rwxrwxrwx 1 root root 0 Sep 6 21:29 f2a -rwxrwxrwx 1 root root 0 Sep 6 21:29 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 6 21:29 f2a -rwxrwxrwx 1 root root 0 Sep 6 21:29 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00944157 s, 54.2 kB/s PASS 2e (10s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 21:29:38 (1788744578) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 21:29:42 (1788744582) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.673257 s, 3.1 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (10s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 21:29:52 (1788744592) PASS 3 (8s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 21:30:00 (1788744600) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 21:30:07 (1788744607) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 21:30:14 (1788744614) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 21:30:21 (1788744621) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 21:30:27 (1788744627) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 21:30:34 (1788744634) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 21:30:42 (1788744642) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780081 s, 13.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0574276 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 21:30:49 (1788744649) 1+0 records in 1+0 records out 1 byte copied, 0.0133217 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00890612 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00912124 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00830698 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0246683 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.028658 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0115126 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0108275 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010481 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0100412 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00938796 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00830141 s, 0.1 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 21:30:58 (1788744658) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0212721 s, 144 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0627017 s, 65.3 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00032588 s, 9.4 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 21:31:06 (1788744666) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 21:31:13 (1788744673) warning: '-runtime' deprecated, use '-t runtime' instead - link 372 (time 1788744685.35 total 10.03 last 37.08) - link 711 (time 1788744695.36 total 20.04 last 33.87) - link 1059 (time 1788744705.39 total 30.07 last 34.69) - link 1385 (time 1788744715.41 total 40.09 last 32.55) - link 1751 (time 1788744725.42 total 50.09 last 36.58) - link 2099 (time 1788744735.42 total 60.09 last 34.80) - link 2450 (time 1788744745.42 total 70.10 last 35.10) - link 2801 (time 1788744755.42 total 80.10 last 35.09) - link 3135 (time 1788744765.42 total 90.10 last 33.38) total: 3390 link in 100.02 seconds: 33.89 ops/second using seed 682204813 running for 100 seconds - stat 5992 (time 1788744687 ; total 11 ; last 11) - stat 10000 (time 1788744693 ; total 17 ; last 6) - stat 15826 (time 1788744704 ; total 28 ; last 11) - stat 20000 (time 1788744711 ; total 35 ; last 7) - stat 26391 (time 1788744722 ; total 46 ; last 11) - stat 30000 (time 1788744728 ; total 52 ; last 6) - stat 33387 (time 1788744739 ; total 63 ; last 11) - stat 37071 (time 1788744750 ; total 74 ; last 11) - stat 40000 (time 1788744759 ; total 83 ; last 9) - stat 42973 (time 1788744770 ; total 94 ; last 11) total: 44455 stats in 100 seconds: 444.549988 stats/second - unlinked 0 (time 1788744726 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3390) error: No such file or directory total: 3389 unlinks in 85 seconds: 39.870586 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21455) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21457) - No such process PASS 12 (144s) == sanityn test 13: test directory page revocation ======= 21:33:36 (1788744816) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 21:33:45 (1788744825) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5139: 22641 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 21:33:51 (1788744831) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5139: 23220 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 21:33:58 (1788744838) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23799 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 21:34:05 (1788744845) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24381 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 21:34:12 (1788744852) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24963 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 21:34:21 (1788744861) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (52s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 21:35:14 (1788744914) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.335981 s, 25.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.318262 s, 26.4 MB/s Chance of close/open is 1 in 50 Seed set to 4918 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x192dc0 000001[1] 1788744919.253308 trunc from 00000000 to 0x192dbf (0x192dc0 bytes) 000002[0] 1788744919.364735 mapwrite 0x6d341b thru 0x6de8e8 (0xb4ce bytes) 000004[1] 1788744919.615581 mapread 0x52c7e0 thru 0x52f61e (0x2e3f bytes) 000005[0] 1788744919.668025 write 0x4ecf3f thru 0x4f016a (0x322c bytes) 000006[1] 1788744919.738680 read 0x5a9daf thru 0x5b69ff (0xcc51 bytes) 000007[1] 1788744919.776354 mapread 0x3f2de0 thru 0x3f4aea (0x1d0b bytes) 000008[0] 1788744919.808733 punch from 0x66814e to 0x669d7f (0x1c32 bytes) 000016[1] 1788744919.868979 punch from 0x11bc6d to 0x12a756 (0xeaea bytes) 000017[0] 1788744919.908205 read 0x1e3cb2 thru 0x1ed47d (0x97cc bytes) 000018[0] 1788744919.931291 punch from 0x637e9c to 0x63cc1f (0x4d84 bytes) 000020[1] 1788744919.952290 punch from 0x17dff6 to 0x1830e7 (0x50f2 bytes) 000021[1] 1788744919.987778 punch from 0x318c7c to 0x324a95 (0xbe1a bytes) 000024[1] 1788744920.034695 read 0x0c3f26 thru 0x0c4ffe (0x10d9 bytes) 000026[0] 1788744920.069788 punch from 0x2b48bd to 0x2c41f8 (0xf93c bytes) 000027[1] 1788744920.086123 punch from 0x45d8d0 to 0x46600a (0x873b bytes) truncating to largest ever: 0x77c8f0 000028[0] 1788744920.113024 trunc from 0x6de8e9 to 0x77c8ef (0x9e007 bytes) 000029[1] 1788744920.171061 mapwrite 0x827719 thru 0x83122a (0x9b12 bytes) 000031[1] 1788744920.327815 write 0x803a3b thru 0x8137eb (0xfdb1 bytes) 000032[1] 1788744920.372563 trunc from 0x83122b to 0x5c12e2 (0x26ff48 bytes) 000033[0] 1788744920.409653 mapwrite 0x5460c0 thru 0x55273d (0xc67e bytes) 000034[1] 1788744920.543303 trunc from 0x5c12e3 to 0x6e7c18 (0x126936 bytes) 000040[0] 1788744920.619949 trunc from 0x6e7c19 to 0x0fdb9c (0x5ea07c bytes) 000042[1] 1788744920.693807 mapread 0x065c88 thru 0x0717d0 (0xbb49 bytes) 000043[1] 1788744920.750219 read 0x0cecf7 thru 0x0db8a0 (0xcbaa bytes) 000045[0] 1788744920.804602 punch from 0x0213af to 0x02d8e3 (0xc535 bytes) 000046[1] 1788744920.840696 read 0x0691b7 thru 0x06d49f (0x42e9 bytes) 000049[1] 1788744920.890871 mapwrite 0x60480c thru 0x60d417 (0x8c0c bytes) 000050[0] 1788744921.108831 mapwrite 0x4edfbd thru 0x4f7f3d (0x9f81 bytes) 000051[0] 1788744921.229669 write 0x31dfb4 thru 0x3287b4 (0xa801 bytes) 000054[1] 1788744921.300246 read 0x0f2dbe thru 0x0fc7d3 (0x9a16 bytes) 000055[1] 1788744921.335792 read 0x12d2a4 thru 0x13c659 (0xf3b6 bytes) 000056[1] 1788744921.397033 mapread 0x5310d1 thru 0x540d39 (0xfc69 bytes) 000058[0] 1788744921.472551 write 0x80b16b thru 0x8157c2 (0xa658 bytes) 000061[0] 1788744921.528622 read 0x75f7a4 thru 0x7680ea (0x8947 bytes) 000063[0] 1788744921.615018 write 0x12d683 thru 0x135e92 (0x8810 bytes) 000064[0] 1788744921.641002 read 0x2a7d34 thru 0x2ab7cd (0x3a9a bytes) 000066[0] 1788744921.676714 trunc from 0x8157c3 to 0x055b28 (0x7bfc9a bytes) 000068[1] 1788744921.772770 mapread 0x02f3ac thru 0x03d4db (0xe130 bytes) 000072[0] 1788744921.974827 mapread 0x038e59 thru 0x040711 (0x78b9 bytes) 000073[1] 1788744922.021635 trunc from 0x055b29 to 0x0b978a (0x63c62 bytes) truncating to largest ever: 0x9f6418 000074[0] 1788744922.067382 trunc from 0x0b978b to 0x9f6417 (0x93cc8d bytes) 000075[0] 1788744922.114020 read 0x53fc24 thru 0x54a132 (0xa50f bytes) 000076[0] 1788744922.146539 read 0x52c8eb thru 0x52e3c2 (0x1ad8 bytes) 000078[1] 1788744922.197215 mapread 0x723a68 thru 0x72ef91 (0xb52a bytes) 000080[1] 1788744922.269284 read 0x5099ad thru 0x516966 (0xcfba bytes) 000081[0] 1788744922.290040 read 0x64b4e2 thru 0x65acc0 (0xf7df bytes) 000082[0] 1788744922.307304 mapwrite 0x5e680c thru 0x5e6930 (0x125 bytes) 000083[1] 1788744922.397017 punch from 0x0232ce to 0x024787 (0x14ba bytes) 000084[1] 1788744922.427257 trunc from 0x9f6418 to 0x2a47e4 (0x751c33 bytes) 000087[0] 1788744922.546100 punch from 0x1dd47c to 0x1dde68 (0x9ed bytes) 000088[1] 1788744922.571664 trunc from 0x2a47e5 to 0x3a8d70 (0x10458c bytes) 000089[0] 1788744922.634991 trunc from 0x3a8d71 to 0x624f7b (0x27c20b bytes) 000091[1] 1788744922.707994 write 0x59b8bb thru 0x59fed0 (0x4616 bytes) 000092[0] 1788744922.746782 punch from 0x3012cf to 0x304be6 (0x3918 bytes) 000093[1] 1788744922.759572 trunc from 0x624f7c to 0x990eda (0x36bf5f bytes) 000095[0] 1788744922.788357 mapread 0x326b93 thru 0x3291c5 (0x2633 bytes) 000096[1] 1788744922.834345 write 0x1e6b54 thru 0x1f63e3 (0xf890 bytes) 000097[0] 1788744922.884527 write 0x946e17 thru 0x94b556 (0x4740 bytes) 000100[0] 1788744922.954658 mapread 0x5c2a9b thru 0x5cb546 (0x8aac bytes) 000102[0] 1788744923.006915 mapread 0x454bcc thru 0x45693d (0x1d72 bytes) 000103[0] 1788744923.024288 read 0x065ad4 thru 0x06b499 (0x59c6 bytes) 000106[1] 1788744923.047546 mapwrite 0x310ef9 thru 0x31ff29 (0xf031 bytes) 000107[0] 1788744923.169140 mapwrite 0x48a9e6 thru 0x48c344 (0x195f bytes) 000108[0] 1788744923.283959 mapwrite 0x47b0a4 thru 0x481e34 (0x6d91 bytes) 000110[0] 1788744923.395089 trunc from 0x990edb to 0x183bbb (0x80d31f bytes) 000111[1] 1788744923.463575 mapwrite 0x52cc55 thru 0x532513 (0x58bf bytes) 000116[1] 1788744923.605111 mapread 0x2411a2 thru 0x243d3a (0x2b99 bytes) 000118[1] 1788744923.644621 mapread 0x1ee937 thru 0x1f5f2d (0x75f7 bytes) 000120[0] 1788744923.702095 mapread 0x260e71 thru 0x262b01 (0x1c91 bytes) 000122[0] 1788744923.747747 mapwrite 0x5bb697 thru 0x5c0ba2 (0x550c bytes) 000125[0] 1788744923.927088 write 0x02e808 thru 0x02ecfa (0x4f3 bytes) 000130[0] 1788744923.983202 write 0x4c07a1 thru 0x4c236d (0x1bcd bytes) 000132[1] 1788744923.995635 trunc from 0x5c0ba3 to 0x2cee53 (0x2f1d4f bytes) 000134[0] 1788744924.083628 write 0x470c7d thru 0x4768b3 (0x5c37 bytes) 000138[0] 1788744924.160622 punch from 0x12bfce to 0x13bef4 (0xff27 bytes) 000139[1] 1788744924.173227 write 0x770c4c thru 0x77d586 (0xc93b bytes) 000140[1] 1788744924.207839 mapwrite 0x0544f3 thru 0x061b11 (0xd61f bytes) 000142[0] 1788744924.354022 write 0x3354da thru 0x3355cf (0xf6 bytes) 000145[1] 1788744924.406766 trunc from 0x77d587 to 0x9950c5 (0x217b3f bytes) 000147[1] 1788744924.491495 read 0x0b41f3 thru 0x0ba480 (0x628e bytes) 000148[0] 1788744924.510133 mapread 0x264b91 thru 0x2658d6 (0xd46 bytes) 000153[1] 1788744924.575206 read 0x128a31 thru 0x137c60 (0xf230 bytes) 000154[0] 1788744924.585932 read 0x510d89 thru 0x51a62a (0x98a2 bytes) 000155[1] 1788744924.613462 mapwrite 0x525b5d thru 0x528dcd (0x3271 bytes) 000156[1] 1788744924.750209 trunc from 0x9950c6 to 0x31712f (0x67df96 bytes) 000158[0] 1788744924.817319 write 0x4f82a4 thru 0x504b16 (0xc873 bytes) 000160[0] 1788744924.860421 punch from 0x35e28d to 0x36a378 (0xc0ec bytes) 000161[0] 1788744924.902945 write 0x34c983 thru 0x354342 (0x79c0 bytes) 000162[1] 1788744924.935149 write 0x841adf thru 0x8457d3 (0x3cf5 bytes) 000163[0] 1788744924.976235 mapread 0x68aff4 thru 0x692bad (0x7bba bytes) 000165[0] 1788744925.118410 read 0x6c1139 thru 0x6c2c9e (0x1b66 bytes) 000166[0] 1788744925.151295 trunc from 0x8457d4 to 0x142fa6 (0x70282d bytes) 000167[0] 1788744925.195191 trunc from 0x142fa7 to 0x3e435b (0x2a13b5 bytes) 000168[0] 1788744925.238989 read 0x22c9e1 thru 0x239032 (0xc652 bytes) 000169[0] 1788744925.326835 read 0x153fcc thru 0x155dc0 (0x1df5 bytes) 000170[0] 1788744925.349666 write 0x8e4147 thru 0x8f2caf (0xeb69 bytes) 000171[1] 1788744925.353990 write 0x73ae7c thru 0x7403a8 (0x552d bytes) 000173[1] 1788744925.393756 mapread 0x8b90c8 thru 0x8bb32b (0x2264 bytes) 000176[0] 1788744925.476235 punch from 0x6cb65e to 0x6d635e (0xad01 bytes) 000179[1] 1788744925.503685 punch from 0x6a7ace to 0x6b51f2 (0xd725 bytes) 000183[0] 1788744925.532417 trunc from 0x8f2cb0 to 0x0f99a9 (0x7f9306 bytes) 000186[0] 1788744925.617655 mapwrite 0x9ebb7e thru 0x9f6dd9 (0xb25c bytes) 000187[0] 1788744925.872478 mapwrite 0x94caec thru 0x95b0d2 (0xe5e7 bytes) 000188[1] 1788744925.942284 mapwrite 0x680423 thru 0x684798 (0x4376 bytes) 000190[0] 1788744926.066960 punch from 0x3c7889 to 0x3d09c3 (0x913b bytes) 000192[0] 1788744926.113608 mapwrite 0x369613 thru 0x371a11 (0x83ff bytes) 000194[1] 1788744926.202171 write 0x5722a5 thru 0x57bd07 (0x9a63 bytes) 000197[0] 1788744926.253580 read 0x909987 thru 0x90d0dc (0x3756 bytes) 000198[1] 1788744926.265831 write 0x85b7c3 thru 0x8689b4 (0xd1f2 bytes) 000200[1] 1788744926.299139 mapread 0x0b62b6 thru 0x0b7709 (0x1454 bytes) 000202[1] 1788744926.330922 write 0x60dac8 thru 0x613485 (0x59be bytes) 000204[1] 1788744926.384978 read 0x89dba9 thru 0x8a9342 (0xb79a bytes) 000206[1] 1788744926.401619 punch from 0x959972 to 0x95ce68 (0x34f7 bytes) 000208[1] 1788744926.424790 write 0x4fd17c thru 0x5010aa (0x3f2f bytes) 000210[1] 1788744926.434598 read 0x54bf90 thru 0x55030a (0x437b bytes) 000211[0] 1788744926.442795 punch from 0x2a5f1c to 0x2a8fb4 (0x3099 bytes) 000212[1] 1788744926.477539 trunc from 0x9f6dda to 0x50d0c4 (0x4e9d15 bytes) 000214[0] 1788744926.557000 write 0x4b7fd8 thru 0x4bf151 (0x717a bytes) 000216[0] 1788744926.609763 punch from 0x4cfb16 to 0x4d886f (0x8d5a bytes) 000218[1] 1788744926.653445 read 0x45e7f0 thru 0x467a0e (0x921f bytes) 000219[1] 1788744926.685291 punch from 0x27cae4 to 0x27fcdd (0x31fa bytes) 000220[0] 1788744926.692610 mapread 0x309a48 thru 0x315ac3 (0xc07c bytes) 000222[1] 1788744926.734688 trunc from 0x50d0c5 to 0x938023 (0x42af5f bytes) 000223[1] 1788744926.787116 write 0x804336 thru 0x80d166 (0x8e31 bytes) 000225[0] 1788744926.797288 read 0x91fb55 thru 0x92cba3 (0xd04f bytes) 000229[1] 1788744926.869716 mapwrite 0x0a7407 thru 0x0b169b (0xa295 bytes) 000230[1] 1788744927.014056 read 0x24461c thru 0x244e3f (0x824 bytes) 000232[1] 1788744927.053093 read 0x81177f thru 0x81ae39 (0x96bb bytes) 000233[0] 1788744927.106782 write 0x5a3c85 thru 0x5ae7f9 (0xab75 bytes) 000234[0] 1788744927.187379 punch from 0x1eb4dc to 0x1eb8bf (0x3e4 bytes) 000236[1] 1788744927.274691 read 0x00d160 thru 0x00e985 (0x1826 bytes) 000238[1] 1788744927.323172 punch from 0x07e7ce to 0x081713 (0x2f46 bytes) 000241[1] 1788744927.410085 mapwrite 0x916f3f thru 0x91c26c (0x532e bytes) 000242[1] 1788744927.594006 mapread 0x0640ec thru 0x065806 (0x171b bytes) 000244[0] 1788744927.700946 write 0x7da997 thru 0x7e3fc1 (0x962b bytes) 000245[1] 1788744927.778134 trunc from 0x938024 to 0x5fa377 (0x33dcac bytes) 000246[0] 1788744927.962655 write 0x879c3d thru 0x87dd4b (0x410f bytes) 000247[1] 1788744928.053072 read 0x1ea975 thru 0x1ee76b (0x3df7 bytes) 000248[1] 1788744928.096606 mapread 0x03cbd8 thru 0x04cb43 (0xff6c bytes) 000249[0] 1788744928.126610 trunc from 0x87dd4c to 0x08b85a (0x7f24f1 bytes) 000253[1] 1788744928.230974 punch from 0x0049d8 to 0x0111d4 (0xc7fd bytes) 000255[0] 1788744928.277725 mapwrite 0x052181 thru 0x059a2e (0x78ae bytes) 000256[0] 1788744928.373796 trunc from 0x08b85b to 0x82a560 (0x79ed06 bytes) 000258[0] 1788744928.425582 trunc from 0x82a561 to 0x7a06e9 (0x89e77 bytes) 000259[0] 1788744928.445772 read 0x3808ef thru 0x38a232 (0x9944 bytes) 000261[0] 1788744928.455007 trunc from 0x7a06ea to 0x1eaeba (0x5b582f bytes) 000262[0] 1788744928.488724 trunc from 0x1eaebb to 0x6b7e03 (0x4ccf49 bytes) 000266[1] 1788744928.571272 read 0x690e63 thru 0x69ee6e (0xe00c bytes) 000268[1] 1788744928.614576 punch from 0x567737 to 0x56e4f0 (0x6dba bytes) 000269[1] 1788744928.640683 mapwrite 0x5b96ee thru 0x5c8f0e (0xf821 bytes) 000270[1] 1788744928.729272 write 0x86d758 thru 0x8767fd (0x90a6 bytes) 000271[0] 1788744928.745371 trunc from 0x8767fe to 0x4a60bf (0x3d073e bytes) 000272[1] 1788744928.798845 mapread 0x382e72 thru 0x390c57 (0xdde6 bytes) 000273[1] 1788744928.893536 mapread 0x3b957a thru 0x3c1628 (0x80af bytes) 000274[0] 1788744928.934013 trunc from 0x4a60c0 to 0x3f98b3 (0xac80c bytes) 000276[0] 1788744928.972411 write 0x0d9862 thru 0x0e0d9c (0x753b bytes) 000277[0] 1788744928.982061 trunc from 0x3f98b4 to 0x4e4a8c (0xeb1d9 bytes) 000279[0] 1788744928.997144 mapwrite 0x52a74e thru 0x52e659 (0x3f0c bytes) 000280[0] 1788744929.090520 mapread 0x227a67 thru 0x22eacf (0x7069 bytes) 000281[0] 1788744929.118794 write 0x2d2350 thru 0x2db099 (0x8d4a bytes) 000282[0] 1788744929.129043 mapread 0x073371 thru 0x07822b (0x4ebb bytes) 000283[0] 1788744929.147171 write 0x0e6bc3 thru 0x0ee020 (0x745e bytes) 000285[0] 1788744929.203728 mapwrite 0x82fa0d thru 0x831759 (0x1d4d bytes) 000287[0] 1788744929.397226 mapwrite 0x004ee2 thru 0x007bd0 (0x2cef bytes) 000288[0] 1788744929.498585 punch from 0x5fdcd3 to 0x6080dc (0xa40a bytes) 000289[0] 1788744929.507779 mapread 0x35af3a thru 0x3645fd (0x96c4 bytes) 000291[0] 1788744929.584823 punch from 0x3a2576 to 0x3ade26 (0xb8b1 bytes) 000292[1] 1788744929.615011 mapread 0x290581 thru 0x298fb1 (0x8a31 bytes) 000293[1] 1788744929.667085 write 0x662bdf thru 0x66a070 (0x7492 bytes) 000298[0] 1788744929.732506 read 0x613f98 thru 0x6224ce (0xe537 bytes) 000299[1] 1788744929.767237 mapread 0x6d6c4d thru 0x6e39fc (0xcdb0 bytes) 000301[1] 1788744929.796676 trunc from 0x83175a to 0x90f316 (0xddbbd bytes) 000302[1] 1788744929.855504 punch from 0x6d586c to 0x6de838 (0x8fcd bytes) 000303[0] 1788744929.859675 mapwrite 0x7852ae thru 0x7888c3 (0x3616 bytes) 000307[1] 1788744930.049178 write 0x8e86a1 thru 0x8ee5c4 (0x5f24 bytes) 000312[1] 1788744930.137630 trunc from 0x90f317 to 0x6b3c1f (0x25b6f7 bytes) 000313[1] 1788744930.189245 read 0x05eed5 thru 0x0693e9 (0xa515 bytes) 000314[0] 1788744930.240351 mapwrite 0x1a45e4 thru 0x1a565f (0x107c bytes) 000316[0] 1788744930.349284 mapread 0x60b562 thru 0x613442 (0x7ee1 bytes) 000317[0] 1788744930.402261 mapwrite 0x845b0d thru 0x84e38b (0x887f bytes) 000318[0] 1788744930.516376 write 0x24005a thru 0x24fd57 (0xfcfe bytes) 000320[0] 1788744930.529376 punch from 0x40f3f0 to 0x418674 (0x9285 bytes) 000321[1] 1788744930.557810 mapread 0x2df727 thru 0x2e2c65 (0x353f bytes) 000322[1] 1788744930.612365 punch from 0x33467e to 0x3416cf (0xd052 bytes) 000323[0] 1788744930.652001 punch from 0x54a74b to 0x55731d (0xcbd3 bytes) 000324[0] 1788744930.673497 mapwrite 0x4b1654 thru 0x4ba9df (0x938c bytes) 000326[0] 1788744930.782197 trunc from 0x84e38c to 0x445570 (0x408e1b bytes) 000327[0] 1788744930.846213 mapread 0x118f0e thru 0x11bb53 (0x2c46 bytes) 000329[0] 1788744930.894848 mapwrite 0x2518f2 thru 0x25f49f (0xdbae bytes) 000330[0] 1788744930.951632 mapwrite 0x248e57 thru 0x2582a8 (0xf452 bytes) 000331[0] 1788744931.000470 read 0x370542 thru 0x37b4e8 (0xafa7 bytes) 000333[1] 1788744931.030797 punch from 0x2a7256 to 0x2af049 (0x7df4 bytes) 000334[0] 1788744931.069782 write 0x09b332 thru 0x0a8bfa (0xd8c9 bytes) 000336[1] 1788744931.127520 mapread 0x110f83 thru 0x11fce7 (0xed65 bytes) 000337[0] 1788744931.188808 read 0x13f7d2 thru 0x140457 (0xc86 bytes) 000342[1] 1788744931.251002 trunc from 0x445571 to 0x8991d5 (0x453c65 bytes) 000344[1] 1788744931.338577 mapwrite 0x633433 thru 0x6374bf (0x408d bytes) 000345[0] 1788744931.418729 mapread 0x6ad7d8 thru 0x6b1c5b (0x4484 bytes) 000348[1] 1788744931.501683 write 0x384b99 thru 0x3897a1 (0x4c09 bytes) 000350[1] 1788744931.526130 punch from 0x19a7bc to 0x1a8200 (0xda45 bytes) 000352[0] 1788744931.543856 mapwrite 0x1b77b8 thru 0x1c546c (0xdcb5 bytes) 000353[1] 1788744931.659016 read 0x117300 thru 0x120ea5 (0x9ba6 bytes) 000354[1] 1788744931.682499 trunc from 0x8991d6 to 0x1bcd80 (0x6dc455 bytes) 000355[0] 1788744931.755686 read 0x040bb0 thru 0x04547c (0x48cd bytes) 000356[1] 1788744931.807018 write 0x2c28c5 thru 0x2caffd (0x8739 bytes) 000357[0] 1788744931.852913 punch from 0x1f2de4 to 0x1fd706 (0xa923 bytes) 000359[1] 1788744931.908293 trunc from 0x2caffe to 0x19a4c5 (0x130b38 bytes) 000360[1] 1788744931.941633 mapwrite 0x7419f9 thru 0x74347e (0x1a86 bytes) 000361[0] 1788744932.037959 write 0x09ba54 thru 0x0a3cf8 (0x82a5 bytes) 000363[1] 1788744932.128023 mapread 0x133ca7 thru 0x140803 (0xcb5d bytes) 000364[0] 1788744932.186290 mapread 0x6fe97b thru 0x701e8d (0x3513 bytes) 000365[1] 1788744932.271345 trunc from 0x74347f to 0x5cdb02 (0x17597c bytes) 000366[1] 1788744932.329141 read 0x3b3133 thru 0x3b7f1c (0x4dea bytes) 000367[1] 1788744932.371581 mapwrite 0x8248f2 thru 0x8338a8 (0xefb7 bytes) 000368[0] 1788744932.477447 mapwrite 0x279d15 thru 0x27e064 (0x4350 bytes) 000369[1] 1788744932.603157 mapwrite 0x59db46 thru 0x5a6eba (0x9375 bytes) 000370[0] 1788744932.725847 punch from 0x2a55e1 to 0x2b14fe (0xbf1e bytes) 000371[0] 1788744932.751620 write 0x78ade7 thru 0x792fa7 (0x81c1 bytes) 000372[0] 1788744932.794014 trunc from 0x8338a9 to 0x7c35d5 (0x702d3 bytes) 000373[1] 1788744932.840518 read 0x2288ae thru 0x22c37b (0x3ace bytes) 000375[1] 1788744932.881193 mapread 0x5432b6 thru 0x551153 (0xde9e bytes) 000377[1] 1788744932.938422 mapread 0x5d253c thru 0x5dc3ad (0x9e72 bytes) 000378[0] 1788744932.959944 mapwrite 0x441fd0 thru 0x44fd4a (0xdd7b bytes) 000379[0] 1788744933.114951 punch from 0x099c99 to 0x0a17a8 (0x7b10 bytes) 000380[1] 1788744933.165254 mapread 0x77e51b thru 0x78cfb6 (0xea9c bytes) 000381[0] 1788744933.250062 mapwrite 0x9e30f8 thru 0x9f2b69 (0xfa72 bytes) 000382[1] 1788744933.491773 trunc from 0x9f2b6a to 0x0ac2b6 (0x9468b3 bytes) 000383[1] 1788744933.547814 punch from 0x03be3e to 0x043a42 (0x7c05 bytes) 000385[1] 1788744933.594799 punch from 0x05597b to 0x0584ca (0x2b50 bytes) 000386[0] 1788744933.672296 trunc from 0x0ac2b7 to 0x9cccf0 (0x920a3a bytes) 000387[0] 1788744933.790289 mapwrite 0x0a399c thru 0x0b2a8f (0xf0f4 bytes) 000397[0] 1788744934.085813 punch from 0x442bcb to 0x44f29a (0xc6d0 bytes) 000398[0] 1788744934.139405 read 0x54f462 thru 0x556d0e (0x78ad bytes) 000399[0] 1788744934.165606 write 0x87e0e9 thru 0x88a480 (0xc398 bytes) 000400[1] 1788744934.192704 punch from 0x5cf441 to 0x5dea9e (0xf65e bytes) 000401[0] 1788744934.213597 trunc from 0x9cccf1 to 0x9eb8d9 (0x1ebe9 bytes) 000402[0] 1788744934.244181 trunc from 0x9eb8da to 0x6fb67e (0x2f025b bytes) 000403[1] 1788744934.282765 trunc from 0x6fb67f to 0x7b85c4 (0xbcf46 bytes) 000404[1] 1788744934.320365 punch from 0x206211 to 0x208e8b (0x2c7b bytes) 000406[1] 1788744934.372052 punch from 0x58917f to 0x593fcc (0xae4e bytes) 000407[1] 1788744934.395817 write 0x5cd260 thru 0x5d6312 (0x90b3 bytes) 000409[1] 1788744934.408060 write 0x525c14 thru 0x5323de (0xc7cb bytes) 000410[0] 1788744934.419073 mapwrite 0x076e70 thru 0x07a145 (0x32d6 bytes) 000411[1] 1788744934.533413 mapwrite 0x11a428 thru 0x120e0e (0x69e7 bytes) 000412[0] 1788744934.663432 read 0x152f30 thru 0x15d71a (0xa7eb bytes) 000413[0] 1788744934.691664 punch from 0x2d8a7a to 0x2e0c16 (0x819d bytes) 000416[0] 1788744934.769837 mapwrite 0x06cc31 thru 0x078b13 (0xbee3 bytes) 000417[1] 1788744934.863879 read 0x21ca44 thru 0x22481d (0x7dda bytes) 000418[1] 1788744934.904756 punch from 0x22cdf1 to 0x239908 (0xcb18 bytes) 000419[1] 1788744934.928162 mapwrite 0x97ba44 thru 0x983232 (0x77ef bytes) 000421[0] 1788744935.090729 punch from 0x4063ae to 0x411371 (0xafc4 bytes) 000423[0] 1788744935.164769 mapwrite 0x72753a thru 0x7363df (0xeea6 bytes) 000425[1] 1788744935.331285 mapread 0x83e2b0 thru 0x83f03a (0xd8b bytes) 000426[0] 1788744935.367993 trunc from 0x983233 to 0x410938 (0x5728fa bytes) 000429[0] 1788744935.478974 mapwrite 0x6be617 thru 0x6c37cc (0x51b6 bytes) 000430[1] 1788744935.594704 write 0x5d375b thru 0x5d976a (0x6010 bytes) 000433[1] 1788744935.671726 mapwrite 0x3f440c thru 0x3fdaf5 (0x96ea bytes) 000434[1] 1788744935.793442 punch from 0x0f233c to 0x0f5849 (0x350e bytes) 000435[0] 1788744935.827390 punch from 0x5fcbff to 0x60ba3d (0xee3f bytes) 000439[0] 1788744935.917867 punch from 0x0d94c9 to 0x0de711 (0x5249 bytes) 000442[0] 1788744935.951465 mapread 0x4e10c2 thru 0x4e550e (0x444d bytes) 000443[1] 1788744936.006653 punch from 0x1cf3d1 to 0x1d9ab1 (0xa6e1 bytes) 000444[0] 1788744936.035043 trunc from 0x6c37cd to 0x08f5dd (0x6341ef bytes) 000445[1] 1788744936.093801 trunc from 0x08f5de to 0x6d372e (0x644151 bytes) 000448[1] 1788744936.162114 mapwrite 0x81bc16 thru 0x829a02 (0xdded bytes) 000449[0] 1788744936.281832 punch from 0x71d838 to 0x72929f (0xba68 bytes) 000452[0] 1788744936.349997 mapread 0x43c403 thru 0x44a164 (0xdd62 bytes) 000455[0] 1788744936.442821 write 0x2bd91a thru 0x2c536f (0x7a56 bytes) 000457[1] 1788744936.473049 mapread 0x5c659e thru 0x5c6d3f (0x7a2 bytes) 000458[1] 1788744936.510715 mapwrite 0x4b061c thru 0x4b2654 (0x2039 bytes) 000462[1] 1788744936.664254 read 0x41e137 thru 0x42c507 (0xe3d1 bytes) 000463[0] 1788744936.691403 write 0x82b75b thru 0x8392c6 (0xdb6c bytes) 000464[1] 1788744936.704516 mapread 0x34037b thru 0x348897 (0x851d bytes) 000466[0] 1788744936.759114 write 0x5b8fae thru 0x5c4601 (0xb654 bytes) 000467[0] 1788744936.791016 trunc from 0x8392c7 to 0x4cfbf6 (0x3696d0 bytes) 000468[0] 1788744936.848966 write 0x7c45fa thru 0x7ca1c2 (0x5bc9 bytes) 000469[0] 1788744936.868374 mapwrite 0x7731dc thru 0x77453e (0x1363 bytes) 000473[0] 1788744936.994818 read 0x5257e6 thru 0x526220 (0xa3b bytes) 000475[1] 1788744937.026765 mapread 0x646f9b thru 0x654b24 (0xdb8a bytes) 000476[0] 1788744937.098360 read 0x3a56dc thru 0x3ae0ec (0x8a11 bytes) 000477[0] 1788744937.119094 write 0x4ef4e2 thru 0x4f4017 (0x4b36 bytes) 000478[1] 1788744937.137664 mapwrite 0x5779e3 thru 0x587919 (0xff37 bytes) 000483[1] 1788744937.311080 write 0x704118 thru 0x70b084 (0x6f6d bytes) 000484[0] 1788744937.332810 mapread 0x52db6f thru 0x5366dd (0x8b6f bytes) 000487[1] 1788744937.378859 read 0x3ee071 thru 0x3fb530 (0xd4c0 bytes) 000488[0] 1788744937.402027 mapread 0x56c8c0 thru 0x56cc0b (0x34c bytes) 000491[1] 1788744937.413713 mapread 0x7b6a0b thru 0x7c1302 (0xa8f8 bytes) 000492[0] 1788744937.439824 read 0x3343e6 thru 0x33fcb7 (0xb8d2 bytes) 000493[1] 1788744937.448562 trunc from 0x7ca1c3 to 0x2b6c60 (0x513562 bytes) 000497[1] 1788744937.593073 read 0x1493dc thru 0x158b73 (0xf798 bytes) 000498[0] 1788744937.650822 punch from 0x103a7d to 0x103eae (0x432 bytes) 000499[1] 1788744937.687025 punch from 0x19a71e to 0x1a681d (0xc100 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4937 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004[1] 1788744938.279453 mapwrite 0x3e0000 thru 0x3e9b46 (0x9b47 bytes) 000005[0] 1788744938.403793 write_OD 0x721000 thru 0x726fff (0x6000 bytes) 000006[1] 1788744938.454045 punch from 0x09ca85 to 0x0a0c41 (0x41bd bytes) 000007[0] 1788744938.473595 mapwrite 0x5f0000 thru 0x5f6339 (0x633a bytes) truncating to largest ever: 0x80e918 000009[1] 1788744938.569616 trunc from 0x727000 to 0x80e917 (0xe7918 bytes) 000010[1] 1788744938.623771 read 0x50e000 thru 0x5165fe (0x85ff bytes) 000017[0] 1788744938.670245 mapwrite 0x118000 thru 0x1199f7 (0x19f8 bytes) 000018[0] 1788744938.760537 trunc from 0x80e918 to 0x611cf8 (0x1fcc1f bytes) 000022[1] 1788744938.870277 punch from 0x2a9d1e to 0x2b9b91 (0xfe74 bytes) 000025[0] 1788744938.891819 punch from 0x01d7ba to 0x02b238 (0xda7f bytes) 000034[1] 1788744938.965806 trunc from 0x611cf9 to 0x58250d (0x8f7eb bytes) 000036[1] 1788744939.012485 trunc from 0x58250e to 0x0c883e (0x4b9ccf bytes) 000037[0] 1788744939.031476 punch from 0x073f3a to 0x07501c (0x10e3 bytes) 000039[1] 1788744939.054688 write 0x201000 thru 0x202050 (0x1051 bytes) 000042[1] 1788744939.097325 trunc from 0x202051 to 0x077dbf (0x18a291 bytes) 000043[1] 1788744939.123697 mapwrite 0x7b1000 thru 0x7b844d (0x744e bytes) 000044[0] 1788744939.198526 mapwrite 0x06c000 thru 0x06fc6e (0x3c6f bytes) 000045[1] 1788744939.269604 mapread 0x3fe000 thru 0x3ffc44 (0x1c45 bytes) 000046[1] 1788744939.294556 trunc from 0x7b844e to 0x6b1a3b (0x106a12 bytes) 000047[1] 1788744939.338555 mapwrite 0x0eb000 thru 0x0f9216 (0xe217 bytes) 000052[1] 1788744939.452541 mapread 0x34c000 thru 0x359f5b (0xdf5c bytes) 000053[1] 1788744939.482845 mapread 0x3ca000 thru 0x3cf48d (0x548e bytes) 000054[1] 1788744939.497211 punch from 0x293d15 to 0x2a13fb (0xd6e7 bytes) 000055[0] 1788744939.513718 read_OD 0x6a6000 thru 0x6affff (0xa000 bytes) 000056[0] 1788744939.538058 trunc from 0x6b1a3c to 0x333ab2 (0x37df89 bytes) 000057[1] 1788744939.567361 write 0x467000 thru 0x473160 (0xc161 bytes) 000059[0] 1788744939.597221 read_OD 0x351000 thru 0x352fff (0x2000 bytes) 000062[0] 1788744939.631624 mapread 0x273000 thru 0x2732ed (0x2ee bytes) 000063[1] 1788744939.636721 read 0x337000 thru 0x33edd7 (0x7dd8 bytes) 000064[0] 1788744939.643255 mapwrite 0x517000 thru 0x51db9b (0x6b9c bytes) 000065[0] 1788744939.763795 trunc from 0x51db9c to 0x04ca36 (0x4d1165 bytes) 000069[0] 1788744939.825352 trunc from 0x04ca37 to 0x686815 (0x639ddf bytes) 000070[1] 1788744939.863544 trunc from 0x686816 to 0x6da436 (0x53c21 bytes) 000071[1] 1788744939.899385 trunc from 0x6da437 to 0x15f7d8 (0x57ac5e bytes) 000072[1] 1788744939.919618 mapread 0x06b000 thru 0x0719ba (0x69bb bytes) 000073[1] 1788744939.958451 punch from 0x03f202 to 0x0466b2 (0x74b1 bytes) 000074[1] 1788744939.962946 write 0x8f4000 thru 0x8f5101 (0x1102 bytes) 000075[0] 1788744939.968634 mapwrite 0x6e9000 thru 0x6f3389 (0xa38a bytes) 000078[0] 1788744940.100758 punch from 0x41b394 to 0x429188 (0xddf5 bytes) 000079[1] 1788744940.118647 mapread 0x3c9000 thru 0x3d88dc (0xf8dd bytes) 000080[0] 1788744940.171160 mapwrite 0x2a6000 thru 0x2b5ad2 (0xfad3 bytes) 000082[1] 1788744940.285536 trunc from 0x8f5102 to 0x340180 (0x5b4f81 bytes) 000083[1] 1788744940.332836 trunc from 0x340181 to 0x78cc3e (0x44cabe bytes) 000087[0] 1788744940.351545 punch from 0x67f0a8 to 0x68c97f (0xd8d8 bytes) 000090[0] 1788744940.419472 mapread 0x2ba000 thru 0x2c5355 (0xb356 bytes) 000092[0] 1788744940.465553 read_OD 0x0a7000 thru 0x0b3fff (0xd000 bytes) 000093[1] 1788744940.490021 mapread 0x37c000 thru 0x37f009 (0x300a bytes) 000094[1] 1788744940.505945 trunc from 0x78cc3f to 0x2b0b5c (0x4dc0e2 bytes) 000095[0] 1788744940.556576 mapread 0x04c000 thru 0x056b8c (0xab8d bytes) 000097[0] 1788744940.602831 read_OD 0x21f000 thru 0x22cfff (0xe000 bytes) 000098[0] 1788744940.611713 write_OD 0x086000 thru 0x090fff (0xb000 bytes) 000099[0] 1788744940.658644 punch from 0x012a7e to 0x01b5e1 (0x8b64 bytes) 000101[0] 1788744940.684957 punch from 0x27765e to 0x284a76 (0xd419 bytes) 000103[0] 1788744940.712976 punch from 0x1bd9fe to 0x1c4e2d (0x7430 bytes) 000106[0] 1788744940.732988 write_OD 0x73a000 thru 0x742fff (0x9000 bytes) 000107[0] 1788744940.785633 mapwrite 0x1ad000 thru 0x1b4de0 (0x7de1 bytes) 000109[0] 1788744940.923746 read_OD 0x509000 thru 0x512fff (0xa000 bytes) 000110[1] 1788744940.953856 write 0x95b000 thru 0x962197 (0x7198 bytes) 000111[0] 1788744940.966753 read_OD 0x6fc000 thru 0x703fff (0x8000 bytes) 000112[0] 1788744941.002796 mapread 0x6af000 thru 0x6b577e (0x677f bytes) 000116[1] 1788744941.030056 punch from 0x64ddd1 to 0x655458 (0x7688 bytes) 000117[1] 1788744941.046059 mapwrite 0x916000 thru 0x921979 (0xb97a bytes) 000118[0] 1788744941.151294 mapwrite 0x4b8000 thru 0x4c2c96 (0xac97 bytes) 000119[0] 1788744941.233089 punch from 0x5a5cc5 to 0x5a9515 (0x3851 bytes) 000120[1] 1788744941.252756 write 0x865000 thru 0x86d2a8 (0x82a9 bytes) 000122[0] 1788744941.303945 mapread 0x684000 thru 0x68dbc4 (0x9bc5 bytes) 000124[1] 1788744941.353035 write 0x3bc000 thru 0x3bc78c (0x78d bytes) 000125[1] 1788744941.371648 trunc from 0x962198 to 0x4cd5e6 (0x494bb1 bytes) 000126[1] 1788744941.406664 trunc from 0x4cd5e7 to 0x2d365b (0x1f9f8b bytes) 000128[0] 1788744941.446018 mapwrite 0x2cf000 thru 0x2cfbe9 (0xbea bytes) 000129[0] 1788744941.524667 read_OD 0x0e0000 thru 0x0e3fff (0x4000 bytes) 000135[0] 1788744941.548614 read_OD 0x26a000 thru 0x26afff (0x1000 bytes) 000137[1] 1788744941.555717 write 0x6a2000 thru 0x6aab64 (0x8b65 bytes) 000138[0] 1788744941.578990 mapwrite 0x27b000 thru 0x28a860 (0xf861 bytes) 000139[0] 1788744941.748163 mapread 0x21e000 thru 0x22518a (0x718b bytes) 000140[0] 1788744941.764083 mapwrite 0x9dc000 thru 0x9ea072 (0xe073 bytes) 000141[0] 1788744941.943468 mapread 0x1f5000 thru 0x204991 (0xf992 bytes) 000142[1] 1788744941.972424 punch from 0x044758 to 0x04c300 (0x7ba9 bytes) 000144[1] 1788744942.033871 mapread 0x026000 thru 0x034242 (0xe243 bytes) 000145[0] 1788744942.071697 write_OD 0x44c000 thru 0x45afff (0xf000 bytes) 000153[0] 1788744942.128829 mapread 0x719000 thru 0x720362 (0x7363 bytes) 000154[1] 1788744942.163625 mapwrite 0x568000 thru 0x569836 (0x1837 bytes) 000157[0] 1788744942.236160 trunc from 0x9ea073 to 0x787568 (0x262b0a bytes) 000161[1] 1788744942.318451 read 0x0c4000 thru 0x0ccbe8 (0x8be9 bytes) 000164[0] 1788744942.347109 trunc from 0x787569 to 0x7179e6 (0x6fb82 bytes) 000165[0] 1788744942.386101 mapwrite 0x1b5000 thru 0x1bf6d1 (0xa6d2 bytes) 000166[0] 1788744942.493010 mapwrite 0x4fb000 thru 0x50838c (0xd38d bytes) 000170[1] 1788744942.625016 read 0x364000 thru 0x36cb9b (0x8b9c bytes) 000171[0] 1788744942.643645 mapread 0x11e000 thru 0x12a983 (0xc984 bytes) 000173[0] 1788744942.702447 punch from 0x1a0f43 to 0x1a1033 (0xf1 bytes) 000174[1] 1788744942.718709 punch from 0x20d083 to 0x216cda (0x9c58 bytes) 000178[0] 1788744942.772914 write_OD 0x11d000 thru 0x12bfff (0xf000 bytes) 000179[1] 1788744942.825709 punch from 0x3fb75d to 0x40399e (0x8242 bytes) 000181[1] 1788744942.853545 mapwrite 0x853000 thru 0x856682 (0x3683 bytes) 000182[1] 1788744942.926506 trunc from 0x856683 to 0x3b3ed6 (0x4a27ac bytes) 000184[1] 1788744942.961653 punch from 0x01d3af to 0x01d7de (0x430 bytes) 000185[0] 1788744942.986723 trunc from 0x3b3ed7 to 0x5a30b5 (0x1ef1df bytes) 000186[0] 1788744943.032392 read_OD 0x414000 thru 0x415fff (0x2000 bytes) 000187[1] 1788744943.044289 mapwrite 0x557000 thru 0x560f37 (0x9f38 bytes) 000188[1] 1788744943.113130 mapread 0x41d000 thru 0x42bdf6 (0xedf7 bytes) 000189[1] 1788744943.138593 mapwrite 0x279000 thru 0x281331 (0x8332 bytes) truncating to largest ever: 0x907860 000193[0] 1788744943.234350 trunc from 0x5a30b6 to 0x90785f (0x3647aa bytes) 000196[0] 1788744943.318998 punch from 0x47bf43 to 0x481498 (0x5556 bytes) 000197[0] 1788744943.336800 mapread 0x2c8000 thru 0x2c8e15 (0xe16 bytes) 000198[1] 1788744943.345111 trunc from 0x907860 to 0x8ee2cc (0x19593 bytes) 000200[1] 1788744943.395802 read 0x071000 thru 0x07115f (0x160 bytes) 000201[1] 1788744943.401729 punch from 0x22c3e2 to 0x238d18 (0xc937 bytes) 000202[1] 1788744943.406563 trunc from 0x8ee2cd to 0x59dd19 (0x3505b3 bytes) 000203[0] 1788744943.436421 mapwrite 0x05a000 thru 0x06175c (0x775d bytes) 000204[1] 1788744943.528759 mapwrite 0x32e000 thru 0x334219 (0x621a bytes) 000206[0] 1788744943.626726 read_OD 0x3d0000 thru 0x3defff (0xf000 bytes) 000207[1] 1788744943.648298 punch from 0x1a4d61 to 0x1ac4b0 (0x7750 bytes) 000208[1] 1788744943.662826 punch from 0x2b29b9 to 0x2bb09f (0x86e7 bytes) 000209[1] 1788744943.677392 write 0x04f000 thru 0x05358a (0x458b bytes) 000210[1] 1788744943.681842 punch from 0x271653 to 0x27c8be (0xb26c bytes) 000218[0] 1788744943.709989 punch from 0x57e52f to 0x5802ef (0x1dc1 bytes) 000220[1] 1788744943.732589 mapread 0x25e000 thru 0x268092 (0xa093 bytes) 000222[0] 1788744943.776529 read_OD 0x2a6000 thru 0x2aefff (0x9000 bytes) 000224[1] 1788744943.784695 mapread 0x165000 thru 0x16fc94 (0xac95 bytes) 000226[1] 1788744943.814520 write 0x00a000 thru 0x010f3a (0x6f3b bytes) 000227[0] 1788744943.847877 mapwrite 0x0a4000 thru 0x0a7ee6 (0x3ee7 bytes) 000228[1] 1788744943.965758 write 0x377000 thru 0x37bb89 (0x4b8a bytes) 000229[1] 1788744943.999474 trunc from 0x59dd1a to 0x4ea195 (0xb3b84 bytes) 000230[0] 1788744944.017554 mapwrite 0x611000 thru 0x615658 (0x4659 bytes) 000231[1] 1788744944.129336 mapread 0x1f4000 thru 0x1ff04c (0xb04d bytes) 000232[0] 1788744944.182421 write_OD 0x369000 thru 0x36dfff (0x5000 bytes) 000234[0] 1788744944.235694 mapwrite 0x57a000 thru 0x5854ed (0xb4ee bytes) 000236[1] 1788744944.343654 read 0x609000 thru 0x611ada (0x8adb bytes) 000239[0] 1788744944.371597 punch from 0x2d29af to 0x2dd1ab (0xa7fd bytes) 000240[1] 1788744944.388584 trunc from 0x615659 to 0x11f5d4 (0x4f6084 bytes) 000242[0] 1788744944.440410 mapwrite 0x214000 thru 0x216c15 (0x2c16 bytes) 000243[1] 1788744944.540034 mapwrite 0x4a1000 thru 0x4aec85 (0xdc86 bytes) 000245[1] 1788744944.654443 read 0x32a000 thru 0x32b87f (0x1880 bytes) 000247[0] 1788744944.659710 mapwrite 0x174000 thru 0x178d15 (0x4d16 bytes) 000248[0] 1788744944.735346 write_OD 0x74d000 thru 0x75bfff (0xf000 bytes) 000249[1] 1788744944.790862 mapwrite 0x1b1000 thru 0x1b1d65 (0xd66 bytes) 000251[1] 1788744944.853537 write 0x82b000 thru 0x834823 (0x9824 bytes) 000252[1] 1788744944.868047 write 0x1c7000 thru 0x1d1fa7 (0xafa8 bytes) 000253[1] 1788744944.884566 punch from 0x483004 to 0x485ec6 (0x2ec3 bytes) 000254[1] 1788744944.900639 mapwrite 0x011000 thru 0x01852d (0x752e bytes) 000257[1] 1788744944.994424 write 0x6a6000 thru 0x6a7ab3 (0x1ab4 bytes) 000259[1] 1788744945.013797 write 0x0c2000 thru 0x0c705d (0x505e bytes) 000261[0] 1788744945.032393 mapread 0x275000 thru 0x284ccd (0xfcce bytes) truncating to largest ever: 0x9531ac 000262[0] 1788744945.090129 trunc from 0x834824 to 0x9531ab (0x11e988 bytes) 000263[0] 1788744945.133472 write_OD 0x031000 thru 0x037fff (0x7000 bytes) 000265[1] 1788744945.182319 read 0x8e4000 thru 0x8eddd2 (0x9dd3 bytes) 000267[0] 1788744945.194362 trunc from 0x9531ac to 0x7f0baa (0x162601 bytes) 000268[1] 1788744945.236263 punch from 0x576c6e to 0x580357 (0x96ea bytes) 000271[0] 1788744945.296987 read_OD 0x2a6000 thru 0x2b2fff (0xd000 bytes) 000272[1] 1788744945.328451 write 0x3cd000 thru 0x3cefd6 (0x1fd7 bytes) 000273[1] 1788744945.356343 mapwrite 0x551000 thru 0x556e97 (0x5e98 bytes) 000274[0] 1788744945.412374 punch from 0x6a9eaf to 0x6b14d8 (0x762a bytes) 000275[1] 1788744945.454809 mapread 0x2e6000 thru 0x2f317d (0xd17e bytes) 000276[1] 1788744945.506753 trunc from 0x7f0bab to 0x38c409 (0x4647a1 bytes) 000277[0] 1788744945.550783 mapwrite 0x1eb000 thru 0x1f370e (0x870f bytes) 000278[1] 1788744945.617811 read 0x2ce000 thru 0x2d4a96 (0x6a97 bytes) 000280[1] 1788744945.685130 write 0x903000 thru 0x9118b0 (0xe8b1 bytes) 000281[0] 1788744945.727487 punch from 0x1f1f12 to 0x1f356d (0x165c bytes) 000282[1] 1788744945.760487 write 0x7c9000 thru 0x7d4fee (0xbfef bytes) 000283[0] 1788744945.863580 read_OD 0x897000 thru 0x89efff (0x8000 bytes) 000288[1] 1788744945.912536 mapwrite 0x3c4000 thru 0x3c500b (0x100c bytes) 000291[0] 1788744945.995553 mapwrite 0x973000 thru 0x97d96f (0xa970 bytes) 000292[1] 1788744946.123156 mapwrite 0x0e3000 thru 0x0ee615 (0xb616 bytes) 000294[0] 1788744946.267940 read_OD 0x191000 thru 0x19afff (0xa000 bytes) 000295[1] 1788744946.301614 read 0x72e000 thru 0x73a440 (0xc441 bytes) 000296[1] 1788744946.319390 trunc from 0x97d970 to 0x26afb6 (0x7129b9 bytes) 000297[0] 1788744946.369229 trunc from 0x26afb7 to 0x5eea73 (0x383abd bytes) 000299[0] 1788744946.401248 read_OD 0x2fb000 thru 0x2fbfff (0x1000 bytes) 000303[1] 1788744946.424982 punch from 0x50ce51 to 0x514140 (0x72f0 bytes) 000304[1] 1788744946.455442 write 0x8a4000 thru 0x8a51e3 (0x11e4 bytes) 000306[0] 1788744946.470556 trunc from 0x8a51e4 to 0x7d5f1f (0xcf2c4 bytes) 000308[1] 1788744946.525718 punch from 0x555054 to 0x555c30 (0xbdd bytes) 000312[0] 1788744946.578114 mapread 0x4df000 thru 0x4e1aee (0x2aef bytes) 000313[0] 1788744946.623648 punch from 0x639e1c to 0x63ce27 (0x300c bytes) 000317[1] 1788744946.704328 mapwrite 0x6f7000 thru 0x6fb39e (0x439f bytes) 000318[1] 1788744946.801561 mapread 0x3eb000 thru 0x3f9cc6 (0xecc7 bytes) 000319[0] 1788744946.834545 punch from 0x0e0b1f to 0x0efea3 (0xf385 bytes) 000323[0] 1788744946.910083 trunc from 0x7d5f20 to 0x0612cd (0x774c52 bytes) 000324[1] 1788744946.964309 mapread 0x009000 thru 0x00e14d (0x514e bytes) 000325[1] 1788744947.030411 mapwrite 0x4dc000 thru 0x4e0635 (0x4636 bytes) 000328[1] 1788744947.173877 mapwrite 0x503000 thru 0x507ff7 (0x4ff8 bytes) 000329[1] 1788744947.266697 mapread 0x2db000 thru 0x2df488 (0x4489 bytes) 000330[1] 1788744947.312359 write 0x1f5000 thru 0x200516 (0xb517 bytes) 000331[0] 1788744947.323489 mapread 0x4e9000 thru 0x4f6825 (0xd826 bytes) 000332[1] 1788744947.374787 punch from 0x0fad93 to 0x0ff2e0 (0x454e bytes) 000333[1] 1788744947.400817 mapread 0x1d3000 thru 0x1d73c1 (0x43c2 bytes) 000334[1] 1788744947.431464 mapread 0x2aa000 thru 0x2b387c (0x987d bytes) 000335[1] 1788744947.467584 punch from 0x05d9a2 to 0x06bf1e (0xe57d bytes) 000337[1] 1788744947.510752 mapwrite 0x824000 thru 0x829131 (0x5132 bytes) 000338[0] 1788744947.593264 mapread 0x808000 thru 0x80ce05 (0x4e06 bytes) 000341[1] 1788744947.631100 punch from 0x27002e to 0x27410e (0x40e1 bytes) 000342[0] 1788744947.660846 mapwrite 0x8c7000 thru 0x8c8a8b (0x1a8c bytes) 000343[1] 1788744947.748994 mapwrite 0x94a000 thru 0x94a5a8 (0x5a9 bytes) 000344[0] 1788744947.825676 mapwrite 0x77e000 thru 0x785f6a (0x7f6b bytes) 000347[0] 1788744947.936084 mapread 0x3cf000 thru 0x3d8784 (0x9785 bytes) 000351[0] 1788744947.986345 punch from 0x66da5b to 0x671e65 (0x440b bytes) 000356[1] 1788744948.043399 punch from 0x0dcfae to 0x0e8435 (0xb488 bytes) 000359[0] 1788744948.083144 mapwrite 0x05a000 thru 0x067f41 (0xdf42 bytes) 000360[0] 1788744948.180619 mapread 0x889000 thru 0x88db41 (0x4b42 bytes) 000362[0] 1788744948.204216 read_OD 0x6bb000 thru 0x6c0fff (0x6000 bytes) 000364[1] 1788744948.224235 mapwrite 0x9e3000 thru 0x9e95c2 (0x65c3 bytes) 000365[0] 1788744948.308283 mapread 0x5ef000 thru 0x5f3ed9 (0x4eda bytes) 000367[0] 1788744948.351595 mapwrite 0x076000 thru 0x083949 (0xd94a bytes) 000369[1] 1788744948.447841 punch from 0x942b09 to 0x94dc8d (0xb185 bytes) 000372[0] 1788744948.489638 mapwrite 0x47d000 thru 0x48279a (0x579b bytes) 000373[1] 1788744948.589459 trunc from 0x9e95c3 to 0x6de7fc (0x30adc6 bytes) 000375[1] 1788744948.623398 trunc from 0x6de7fd to 0x6765e1 (0x6821b bytes) 000376[1] 1788744948.646815 read 0x4cb000 thru 0x4d902b (0xe02c bytes) 000378[1] 1788744948.659838 write 0x42f000 thru 0x430a7e (0x1a7f bytes) 000381[0] 1788744948.701470 read_OD 0x08c000 thru 0x099fff (0xe000 bytes) 000383[0] 1788744948.732291 mapread 0x38b000 thru 0x38e17a (0x317b bytes) 000385[1] 1788744948.752773 mapread 0x49b000 thru 0x4a2441 (0x7442 bytes) 000388[0] 1788744948.776358 mapwrite 0x883000 thru 0x88d8ea (0xa8eb bytes) 000391[0] 1788744948.888112 punch from 0x2a2ce2 to 0x2b0e44 (0xe163 bytes) 000392[1] 1788744948.895837 write 0x6a1000 thru 0x6b0cc9 (0xfcca bytes) 000393[1] 1788744948.922875 write 0x90a000 thru 0x911d07 (0x7d08 bytes) 000394[1] 1788744948.937656 write 0x361000 thru 0x36ecbc (0xdcbd bytes) 000396[0] 1788744948.957244 mapread 0x671000 thru 0x67826e (0x726f bytes) 000397[1] 1788744949.005991 mapwrite 0x1e8000 thru 0x1f42b6 (0xc2b7 bytes) truncating to largest ever: 0x9633c4 000401[1] 1788744949.172474 trunc from 0x911d08 to 0x9633c3 (0x516bc bytes) 000402[0] 1788744949.191741 mapwrite 0x909000 thru 0x91834e (0xf34f bytes) 000403[1] 1788744949.308455 read 0x2cd000 thru 0x2d1819 (0x481a bytes) 000406[1] 1788744949.335740 mapread 0x24a000 thru 0x24e98f (0x4990 bytes) 000407[1] 1788744949.344639 read 0x40b000 thru 0x41900f (0xe010 bytes) 000409[0] 1788744949.353218 punch from 0x4001fe to 0x4091ff (0x9002 bytes) 000410[0] 1788744949.368443 mapread 0x2cb000 thru 0x2d4f40 (0x9f41 bytes) 000411[1] 1788744949.396065 trunc from 0x9633c4 to 0x74217e (0x221245 bytes) 000417[0] 1788744949.465709 read_OD 0x709000 thru 0x70dfff (0x5000 bytes) 000419[0] 1788744949.498017 trunc from 0x74217f to 0x3daa0d (0x367771 bytes) 000420[1] 1788744949.545356 trunc from 0x3daa0e to 0x4aee73 (0xd4466 bytes) 000422[1] 1788744949.601101 read 0x0fa000 thru 0x1002d3 (0x62d4 bytes) 000424[0] 1788744949.628500 mapread 0x137000 thru 0x142123 (0xb124 bytes) 000425[1] 1788744949.692963 write 0x0ef000 thru 0x0f390b (0x490c bytes) 000427[0] 1788744949.746006 read_OD 0x1b4000 thru 0x1b4fff (0x1000 bytes) 000429[1] 1788744949.778605 write 0x4ae000 thru 0x4af21c (0x121d bytes) 000431[1] 1788744949.826781 write 0x006000 thru 0x006459 (0x45a bytes) 000434[1] 1788744949.843886 trunc from 0x4af21d to 0x5ccba0 (0x11d984 bytes) 000437[0] 1788744949.892476 mapread 0x3b0000 thru 0x3b6f70 (0x6f71 bytes) 000439[0] 1788744949.948663 trunc from 0x5ccba1 to 0x03fdf9 (0x58cda7 bytes) 000441[0] 1788744950.001359 write_OD 0x9db000 thru 0x9e6fff (0xc000 bytes) 000450[0] 1788744950.103129 mapread 0x64d000 thru 0x6575d5 (0xa5d6 bytes) 000452[0] 1788744950.126220 mapread 0x816000 thru 0x8244eb (0xe4ec bytes) 000453[0] 1788744950.152707 mapwrite 0x6ea000 thru 0x6f7fba (0xdfbb bytes) 000454[1] 1788744950.270095 write 0x278000 thru 0x28407e (0xc07f bytes) 000455[1] 1788744950.286737 trunc from 0x9e7000 to 0x580c8c (0x466373 bytes) 000457[1] 1788744950.309872 punch from 0x1692a0 to 0x16f11c (0x5e7d bytes) 000459[1] 1788744950.314682 mapread 0x392000 thru 0x39626d (0x426e bytes) 000460[0] 1788744950.358263 mapwrite 0x6ed000 thru 0x6f8012 (0xb013 bytes) 000463[0] 1788744950.474615 punch from 0x424e30 to 0x42f8a1 (0xaa72 bytes) 000464[0] 1788744950.497595 write_OD 0x7a1000 thru 0x7abfff (0xb000 bytes) 000470[0] 1788744950.595813 trunc from 0x7ac000 to 0x178064 (0x633f9b bytes) 000471[0] 1788744950.686646 write_OD 0x196000 thru 0x1a2fff (0xd000 bytes) 000472[1] 1788744950.828551 write 0x726000 thru 0x7320e6 (0xc0e7 bytes) 000474[1] 1788744950.893442 punch from 0x389d2d to 0x3931f2 (0x94c6 bytes) 000476[1] 1788744950.918495 mapwrite 0x393000 thru 0x3985b7 (0x55b8 bytes) 000477[1] 1788744951.067879 trunc from 0x7320e7 to 0x1c6dcd (0x56b319 bytes) 000481[0] 1788744951.158051 mapread 0x064000 thru 0x069df1 (0x5df2 bytes) 000482[0] 1788744951.243976 mapread 0x0ea000 thru 0x0ef1e8 (0x51e9 bytes) 000483[0] 1788744951.258355 trunc from 0x1c6dce to 0x166a80 (0x6034d bytes) 000486[1] 1788744951.339944 read 0x157000 thru 0x15eb5f (0x7b60 bytes) 000487[0] 1788744951.392559 mapread 0x0c5000 thru 0x0c9325 (0x4326 bytes) 000488[0] 1788744951.407799 mapread 0x059000 thru 0x05a747 (0x1748 bytes) 000489[0] 1788744951.419429 mapwrite 0x44f000 thru 0x4548ef (0x58f0 bytes) 000491[1] 1788744951.549425 trunc from 0x4548f0 to 0x677bef (0x223300 bytes) 000493[1] 1788744951.582776 punch from 0x17479b to 0x17a47d (0x5ce3 bytes) 000494[1] 1788744951.603621 trunc from 0x677bf0 to 0x0b6091 (0x5c1b5e bytes) 000498[1] 1788744951.646720 trunc from 0x0b6092 to 0x5106bd (0x45a62c bytes) 000499[1] 1788744951.694486 punch from 0x1b0f4a to 0x1ba6b2 (0x9769 bytes) All operations completed A-OK! PASS 16a (42s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 21:35:57 (1788744957) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.547804 s, 15.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.27821 s, 30.2 MB/s Chance of close/open is 1 in 50 Seed set to 4960 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1788744961.474799 write 0x8f965a thru 0x8fa07a (0xa21 bytes) 000003[0] 1788744961.506289 mapwrite 0x6ec32b thru 0x6ecf24 (0xbfa bytes) 000004[1] 1788744961.635338 mapwrite 0x2c016a thru 0x2c04dd (0x374 bytes) 000005[0] 1788744961.772790 mapwrite 0x87c100 thru 0x87dbbe (0x1abf bytes) 000006[1] 1788744961.937729 mapwrite 0x62250f thru 0x622cb1 (0x7a3 bytes) 000007[1] 1788744962.020949 mapread 0x0a5587 thru 0x0a6b26 (0x15a0 bytes) 000009[1] 1788744962.061037 punch from 0x5e08bc to 0x5e1cbd (0x1402 bytes) 000011[1] 1788744962.084098 punch from 0x03fb2c to 0x041ad8 (0x1fad bytes) truncating to largest ever: 0x6d0ee1 000013[0] 1788744962.141600 trunc from 0x8fa07b to 0x6d0ee0 (0x22919a bytes) 000016[1] 1788744962.241861 punch from 0x17d400 to 0x17de13 (0xa14 bytes) 000018[1] 1788744962.295877 read 0x477710 thru 0x4778f7 (0x1e8 bytes) 000020[1] 1788744962.310628 trunc from 0x6d0ee1 to 0x54c2e6 (0x184bfa bytes) 000021[1] 1788744962.334666 mapwrite 0x0a3777 thru 0x0a4d8c (0x1616 bytes) 000022[1] 1788744962.456173 read 0x1a126a thru 0x1a1515 (0x2ac bytes) 000023[0] 1788744962.462623 mapwrite 0x6e78f7 thru 0x6e7f82 (0x68c bytes) 000024[0] 1788744962.589023 write 0x83a43a thru 0x83b1f3 (0xdba bytes) 000025[0] 1788744962.639767 mapread 0x0cddc6 thru 0x0ce58a (0x7c5 bytes) truncating to largest ever: 0x937fc6 000026[0] 1788744962.674038 trunc from 0x83b1f4 to 0x937fc5 (0xfcdd2 bytes) 000028[0] 1788744962.715792 write 0x556252 thru 0x556381 (0x130 bytes) 000029[0] 1788744962.741683 trunc from 0x937fc6 to 0x570e9b (0x3c712a bytes) 000030[1] 1788744962.770450 trunc from 0x570e9c to 0x8d66da (0x36583f bytes) 000031[0] 1788744962.828605 trunc from 0x8d66db to 0x045413 (0x8912c7 bytes) 000032[0] 1788744962.867398 read 0x00c5b5 thru 0x00c7da (0x226 bytes) 000034[1] 1788744962.872581 read 0x03958c thru 0x03ad7f (0x17f4 bytes) 000035[1] 1788744962.886430 punch from 0x0234ed to 0x024032 (0xb46 bytes) 000036[1] 1788744962.908555 trunc from 0x045414 to 0x6e11b4 (0x69bda1 bytes) 000037[1] 1788744962.935620 mapwrite 0x8d1fcd thru 0x8d38fb (0x192f bytes) 000038[0] 1788744963.023412 trunc from 0x8d38fc to 0x605e1b (0x2cdae0 bytes) 000040[1] 1788744963.079826 mapwrite 0x7da9d3 thru 0x7dacfb (0x329 bytes) 000041[0] 1788744963.218027 mapread 0x764072 thru 0x764fcd (0xf5c bytes) 000042[1] 1788744963.315408 trunc from 0x7dacfc to 0x053819 (0x7874e2 bytes) 000044[1] 1788744963.406467 mapread 0x02898a thru 0x029583 (0xbfa bytes) 000046[1] 1788744963.451510 trunc from 0x05381a to 0x323ce8 (0x2d04cf bytes) 000047[0] 1788744963.473309 punch from 0x2aeb30 to 0x2aee7f (0x350 bytes) 000048[0] 1788744963.535150 mapwrite 0x7c22b9 thru 0x7c280b (0x553 bytes) 000050[1] 1788744963.638521 trunc from 0x7c280c to 0x6060c9 (0x1bc742 bytes) 000051[0] 1788744963.675304 read 0x4745ba thru 0x475497 (0xede bytes) 000052[1] 1788744963.693095 punch from 0x49d0e7 to 0x49ebc1 (0x1adb bytes) 000053[0] 1788744963.706439 read 0x196859 thru 0x196b4c (0x2f4 bytes) 000054[0] 1788744963.729051 mapwrite 0x8c0e5d thru 0x8c25b0 (0x1754 bytes) 000055[1] 1788744963.843422 mapwrite 0x616eb9 thru 0x617fcd (0x1115 bytes) 000057[0] 1788744963.959812 read 0x7e4427 thru 0x7e4c46 (0x820 bytes) 000062[1] 1788744964.017691 write 0x24d46d thru 0x24f035 (0x1bc9 bytes) 000065[0] 1788744964.082778 mapread 0x3e3cac thru 0x3e4aca (0xe1f bytes) 000066[0] 1788744964.122861 mapwrite 0x2b8591 thru 0x2b9fa0 (0x1a10 bytes) 000068[1] 1788744964.245099 punch from 0x4bc0ac to 0x4bcc3f (0xb94 bytes) 000069[1] 1788744964.280384 mapwrite 0x83ef6a thru 0x84067c (0x1713 bytes) 000072[0] 1788744964.408473 read 0x32edd4 thru 0x32fcf3 (0xf20 bytes) 000073[0] 1788744964.432451 punch from 0x5bca88 to 0x5bcadc (0x55 bytes) 000074[0] 1788744964.484702 mapread 0x4bbd30 thru 0x4bd075 (0x1346 bytes) 000075[0] 1788744964.532560 mapread 0x09fbca thru 0x0a0702 (0xb39 bytes) 000076[0] 1788744964.543930 mapwrite 0x920a8e thru 0x921238 (0x7ab bytes) 000077[1] 1788744964.684931 punch from 0x3c6baa to 0x3c724d (0x6a4 bytes) 000078[0] 1788744964.721451 trunc from 0x921239 to 0x63fa77 (0x2e17c1 bytes) 000079[1] 1788744964.763584 mapwrite 0x7e1dae thru 0x7e3a70 (0x1cc3 bytes) 000080[0] 1788744964.846569 mapread 0x092a2a thru 0x0932ef (0x8c6 bytes) 000081[1] 1788744964.898353 read 0x5e96a0 thru 0x5eab2c (0x148d bytes) 000083[1] 1788744964.935580 mapwrite 0x065f0e thru 0x0677a9 (0x189c bytes) 000085[0] 1788744965.026997 punch from 0x468b19 to 0x46a72b (0x1c13 bytes) 000088[0] 1788744965.078237 read 0x55d511 thru 0x55efc3 (0x1ab3 bytes) 000090[0] 1788744965.135848 write 0x0e8c61 thru 0x0ea44c (0x17ec bytes) 000093[0] 1788744965.276944 mapread 0x390798 thru 0x3917f4 (0x105d bytes) 000095[1] 1788744965.326783 read 0x4a2a09 thru 0x4a35f4 (0xbec bytes) 000097[1] 1788744965.392532 write 0x7286cf thru 0x7293f4 (0xd26 bytes) 000099[1] 1788744965.437617 mapwrite 0x79ad39 thru 0x79c701 (0x19c9 bytes) 000100[0] 1788744965.514749 read 0x535928 thru 0x535e55 (0x52e bytes) 000104[1] 1788744965.552062 write 0x1c95ac thru 0x1cb369 (0x1dbe bytes) 000105[1] 1788744965.571387 write 0x5b67d9 thru 0x5b714d (0x975 bytes) 000107[1] 1788744965.613081 trunc from 0x7e3a71 to 0x1764a9 (0x66d5c7 bytes) 000108[1] 1788744965.632560 mapwrite 0x41c4cf thru 0x41d5f2 (0x1124 bytes) 000110[0] 1788744965.733779 read 0x0ce7ec thru 0x0cf553 (0xd68 bytes) 000111[1] 1788744965.754423 punch from 0x169dd1 to 0x16a541 (0x771 bytes) 000113[1] 1788744965.771063 read 0x182d57 thru 0x1838a2 (0xb4c bytes) 000117[1] 1788744965.777602 write 0x94e204 thru 0x94f748 (0x1545 bytes) 000119[1] 1788744965.804120 mapread 0x1abc44 thru 0x1adc0e (0x1fcb bytes) 000120[0] 1788744965.835114 punch from 0x13edb5 to 0x13fc01 (0xe4d bytes) 000122[0] 1788744965.873756 punch from 0x92d8b9 to 0x92e40e (0xb56 bytes) 000123[0] 1788744965.890862 mapwrite 0x6f73fc thru 0x6f78f3 (0x4f8 bytes) 000124[0] 1788744965.967783 mapread 0x01929b thru 0x01abb2 (0x1918 bytes) 000126[1] 1788744966.022335 write 0x73ff0e thru 0x7416f8 (0x17eb bytes) 000127[0] 1788744966.052506 write 0x54bb3c thru 0x54d8bc (0x1d81 bytes) 000128[1] 1788744966.088574 mapread 0x62b15a thru 0x62c785 (0x162c bytes) 000130[0] 1788744966.139377 mapwrite 0x574098 thru 0x575df3 (0x1d5c bytes) 000133[1] 1788744966.224017 read 0x59b8b1 thru 0x59cbd1 (0x1321 bytes) 000135[1] 1788744966.267747 mapwrite 0x650345 thru 0x651488 (0x1144 bytes) 000136[1] 1788744966.347934 write 0x9f606f thru 0x9f6a41 (0x9d3 bytes) 000138[1] 1788744966.359623 punch from 0x213e4f to 0x21462f (0x7e1 bytes) 000139[1] 1788744966.377226 mapwrite 0x04780e thru 0x0486a7 (0xe9a bytes) 000140[1] 1788744966.426434 write 0x12d460 thru 0x12de71 (0xa12 bytes) 000141[1] 1788744966.431252 mapread 0x9f1744 thru 0x9f3303 (0x1bc0 bytes) 000143[0] 1788744966.451219 punch from 0x44c096 to 0x44d659 (0x15c4 bytes) 000144[0] 1788744966.478450 mapread 0x68b97c thru 0x68c667 (0xcec bytes) 000145[0] 1788744966.507601 punch from 0x7d0c9e to 0x7d15ab (0x90e bytes) 000146[1] 1788744966.533714 mapread 0x89a94e thru 0x89c419 (0x1acc bytes) 000148[1] 1788744966.594309 write 0x3171f3 thru 0x31888c (0x169a bytes) 000149[0] 1788744966.625880 read 0x0f9363 thru 0x0fa36c (0x100a bytes) 000150[1] 1788744966.700072 mapwrite 0x5dbee8 thru 0x5dc851 (0x96a bytes) 000151[0] 1788744966.772056 mapwrite 0x07ba63 thru 0x07d25d (0x17fb bytes) 000154[1] 1788744966.898238 mapread 0x670dfb thru 0x6710eb (0x2f1 bytes) 000155[0] 1788744966.931280 mapwrite 0x6f9cfe thru 0x6fa3de (0x6e1 bytes) 000156[0] 1788744967.000998 write 0x5e1bc1 thru 0x5e23bd (0x7fd bytes) 000158[0] 1788744967.034013 trunc from 0x9f6a42 to 0x311dcd (0x6e4c74 bytes) 000159[1] 1788744967.056755 trunc from 0x311dce to 0x269f91 (0xa7e3c bytes) 000160[0] 1788744967.096861 mapwrite 0x7796a5 thru 0x77a102 (0xa5e bytes) 000161[1] 1788744967.212678 read 0x4dc931 thru 0x4ddcef (0x13bf bytes) 000162[0] 1788744967.246354 write 0x5b51e0 thru 0x5b5cc0 (0xae1 bytes) 000164[1] 1788744967.297032 mapread 0x147d17 thru 0x1485e5 (0x8cf bytes) 000166[1] 1788744967.339571 mapwrite 0x68c497 thru 0x68cf25 (0xa8f bytes) 000168[0] 1788744967.441383 write 0x1f0590 thru 0x1f0918 (0x389 bytes) 000169[0] 1788744967.476886 mapwrite 0x55307e thru 0x554155 (0x10d8 bytes) 000170[1] 1788744967.600983 punch from 0x149e87 to 0x14a34f (0x4c9 bytes) 000171[1] 1788744967.644297 punch from 0x4227f1 to 0x423122 (0x932 bytes) 000172[0] 1788744967.660362 write 0x405600 thru 0x40729b (0x1c9c bytes) 000175[0] 1788744967.737053 punch from 0x0e97b6 to 0x0ea5f7 (0xe42 bytes) 000177[1] 1788744967.774383 write 0x1f8863 thru 0x1f9d31 (0x14cf bytes) 000178[1] 1788744967.823343 read 0x2938be thru 0x295382 (0x1ac5 bytes) 000181[1] 1788744967.859195 write 0x0b86b7 thru 0x0b8efd (0x847 bytes) 000182[0] 1788744967.877832 write 0x6934e2 thru 0x6940bc (0xbdb bytes) 000183[0] 1788744967.901189 read 0x71278b thru 0x7134c2 (0xd38 bytes) 000187[0] 1788744967.957016 mapwrite 0x839f4e thru 0x83b335 (0x13e8 bytes) 000188[1] 1788744968.148285 read 0x110ba2 thru 0x112235 (0x1694 bytes) 000189[0] 1788744968.182373 read 0x4a1ae3 thru 0x4a22e2 (0x800 bytes) 000190[1] 1788744968.205003 read 0x21b7c4 thru 0x21cccd (0x150a bytes) 000191[0] 1788744968.210469 mapread 0x258add thru 0x25a535 (0x1a59 bytes) 000194[1] 1788744968.247976 mapread 0x145187 thru 0x1464c3 (0x133d bytes) 000195[1] 1788744968.281115 mapwrite 0x907e9c thru 0x9095a2 (0x1707 bytes) 000196[1] 1788744968.417724 punch from 0x7c9552 to 0x7c966d (0x11c bytes) 000197[0] 1788744968.457306 mapread 0x7de8db thru 0x7e0768 (0x1e8e bytes) truncating to largest ever: 0x9d4370 000198[1] 1788744968.532482 trunc from 0x9095a3 to 0x9d436f (0xcadcd bytes) 000199[0] 1788744968.593491 mapwrite 0x27e4b6 thru 0x27f670 (0x11bb bytes) 000203[1] 1788744968.738693 write 0x79651e thru 0x797e4c (0x192f bytes) 000207[1] 1788744968.809472 punch from 0x7395dc to 0x73a56d (0xf92 bytes) 000208[0] 1788744968.831299 mapread 0x2f8158 thru 0x2f8260 (0x109 bytes) 000209[1] 1788744968.894306 trunc from 0x9d4370 to 0x07dcc8 (0x9566a7 bytes) 000211[1] 1788744968.993382 read 0x03d94a thru 0x03e795 (0xe4c bytes) 000213[0] 1788744969.035768 trunc from 0x07dcc9 to 0x54b7ed (0x4cdb25 bytes) 000214[0] 1788744969.085773 read 0x0ff3a3 thru 0x100d2b (0x1989 bytes) 000216[1] 1788744969.091343 mapwrite 0x7327f1 thru 0x734795 (0x1fa5 bytes) 000217[1] 1788744969.357007 mapwrite 0x9efec7 thru 0x9f0e2b (0xf65 bytes) 000220[0] 1788744969.497414 read 0x3ca5f8 thru 0x3cb2f5 (0xcfe bytes) 000221[0] 1788744969.537242 mapwrite 0x023749 thru 0x023ada (0x392 bytes) 000223[0] 1788744969.645177 trunc from 0x9f0e2c to 0x176bf2 (0x87a239 bytes) 000224[0] 1788744969.797028 mapread 0x0b2c19 thru 0x0b37a7 (0xb8f bytes) 000225[0] 1788744969.919017 write 0x2834a9 thru 0x283fe5 (0xb3d bytes) 000228[0] 1788744969.970873 trunc from 0x283fe6 to 0x136917 (0x14d6ce bytes) 000230[1] 1788744970.100011 trunc from 0x136918 to 0x840eeb (0x70a5d4 bytes) 000231[0] 1788744970.181303 trunc from 0x840eec to 0x7b6f24 (0x89fc7 bytes) 000232[1] 1788744970.220027 write 0x4b25ca thru 0x4b3667 (0x109e bytes) 000234[0] 1788744970.272507 write 0x4bde85 thru 0x4bf204 (0x1380 bytes) 000235[1] 1788744970.330276 read 0x4b5b52 thru 0x4b5e80 (0x32f bytes) 000236[1] 1788744970.375532 trunc from 0x7b6f25 to 0x392b60 (0x4243c4 bytes) 000237[0] 1788744970.454014 mapwrite 0x8284d4 thru 0x82a216 (0x1d43 bytes) 000238[0] 1788744970.667990 mapwrite 0x8d0090 thru 0x8d1bb8 (0x1b29 bytes) 000242[0] 1788744970.753393 write 0x27691b thru 0x2770d6 (0x7bc bytes) 000245[0] 1788744970.784612 mapread 0x655552 thru 0x656aef (0x159e bytes) 000247[0] 1788744970.795074 punch from 0x863c7f to 0x8640f2 (0x474 bytes) 000249[1] 1788744970.803344 write 0x39c652 thru 0x39e5c9 (0x1f78 bytes) 000250[1] 1788744970.872562 punch from 0x20a46e to 0x20b1cb (0xd5e bytes) 000251[0] 1788744970.914469 trunc from 0x8d1bb9 to 0x02f74c (0x8a246c bytes) 000253[0] 1788744971.003159 write 0x3419df thru 0x342fb6 (0x15d8 bytes) 000254[1] 1788744971.061720 read 0x13812c thru 0x1390a7 (0xf7c bytes) 000255[1] 1788744971.174796 punch from 0x1b77e9 to 0x1b7e97 (0x6af bytes) 000257[0] 1788744971.219490 mapread 0x03844d thru 0x03889e (0x452 bytes) 000258[0] 1788744971.241835 punch from 0x01b9d2 to 0x01bf92 (0x5c1 bytes) 000259[1] 1788744971.275026 write 0x4e9dfc thru 0x4ea9fc (0xc01 bytes) 000261[1] 1788744971.336797 mapread 0x1185b7 thru 0x1186c7 (0x111 bytes) 000263[1] 1788744971.382091 mapread 0x307696 thru 0x307a7e (0x3e9 bytes) 000264[1] 1788744971.402023 write 0x94598f thru 0x94603b (0x6ad bytes) 000266[1] 1788744971.454303 mapread 0x6d1e5e thru 0x6d3053 (0x11f6 bytes) 000268[0] 1788744971.517682 mapwrite 0x5f081a thru 0x5f1938 (0x111f bytes) 000270[1] 1788744971.674055 trunc from 0x94603c to 0x9a6c41 (0x60c06 bytes) 000274[1] 1788744971.774454 punch from 0x509488 to 0x509518 (0x91 bytes) 000275[1] 1788744971.779623 read 0x1fe276 thru 0x1fed8e (0xb19 bytes) 000276[0] 1788744971.783301 mapwrite 0x685ed0 thru 0x687cec (0x1e1d bytes) 000277[0] 1788744971.853984 write 0x280d8a thru 0x281282 (0x4f9 bytes) 000278[1] 1788744971.879235 write 0x47512a thru 0x475dde (0xcb5 bytes) 000279[0] 1788744971.908879 mapwrite 0x0b5711 thru 0x0b5ae7 (0x3d7 bytes) 000282[1] 1788744972.011082 mapread 0x51eb56 thru 0x51ed78 (0x223 bytes) 000283[1] 1788744972.035987 punch from 0x998c87 to 0x9999a7 (0xd21 bytes) 000286[1] 1788744972.080572 write 0x1f0e02 thru 0x1f26f7 (0x18f6 bytes) 000287[1] 1788744972.093966 punch from 0x64d39b to 0x64ec6d (0x18d3 bytes) 000288[1] 1788744972.122475 write 0x55feae thru 0x561ccf (0x1e22 bytes) 000289[0] 1788744972.157744 trunc from 0x9a6c42 to 0x5c83d7 (0x3de86a bytes) 000291[1] 1788744972.215656 trunc from 0x5c83d8 to 0x57028f (0x58148 bytes) 000292[1] 1788744972.263003 mapread 0x42054e thru 0x420f92 (0xa45 bytes) 000293[1] 1788744972.267805 mapwrite 0x5ea206 thru 0x5ebdc4 (0x1bbf bytes) 000294[1] 1788744972.338985 trunc from 0x5ebdc5 to 0x7778b9 (0x18baf5 bytes) 000297[0] 1788744972.414577 mapwrite 0x2fb3f8 thru 0x2fd2d0 (0x1ed9 bytes) 000301[0] 1788744972.608243 trunc from 0x7778ba to 0x0290de (0x74e7db bytes) 000302[0] 1788744972.636989 mapread 0x0135cc thru 0x014023 (0xa58 bytes) 000303[0] 1788744972.645867 mapread 0x002c8e thru 0x0036be (0xa31 bytes) 000304[0] 1788744972.652214 mapwrite 0x3d0453 thru 0x3d15b9 (0x1167 bytes) 000305[1] 1788744972.751470 trunc from 0x3d15ba to 0x45b77b (0x8a1c2 bytes) 000306[0] 1788744972.830247 write 0x5104ff thru 0x510e26 (0x928 bytes) 000307[0] 1788744972.893038 read 0x3543b8 thru 0x354748 (0x391 bytes) 000308[0] 1788744972.969452 mapwrite 0x1df8ba thru 0x1e01d9 (0x920 bytes) 000315[0] 1788744973.218847 punch from 0x4f7590 to 0x4f8ee8 (0x1959 bytes) 000318[1] 1788744973.230408 write 0x5d053f thru 0x5d1d51 (0x1813 bytes) 000319[0] 1788744973.305323 write 0x99074e thru 0x99222e (0x1ae1 bytes) 000322[1] 1788744973.378463 mapread 0x45c402 thru 0x45c573 (0x172 bytes) 000323[1] 1788744973.420664 write 0x0c19f8 thru 0x0c1bd8 (0x1e1 bytes) 000324[0] 1788744973.457952 write 0x4273e6 thru 0x427fd6 (0xbf1 bytes) 000325[1] 1788744973.503769 mapread 0x962f6f thru 0x963332 (0x3c4 bytes) 000329[0] 1788744973.602230 write 0x24348e thru 0x24524c (0x1dbf bytes) 000330[0] 1788744973.628861 punch from 0x1adedc to 0x1af0d7 (0x11fc bytes) 000331[0] 1788744973.659282 mapread 0x865b09 thru 0x867100 (0x15f8 bytes) 000333[1] 1788744973.679721 read 0x4e10c4 thru 0x4e2c3b (0x1b78 bytes) 000334[1] 1788744973.705841 trunc from 0x99222f to 0x1e811e (0x7aa110 bytes) 000335[0] 1788744973.795475 write 0x77de81 thru 0x77e0e5 (0x265 bytes) 000337[1] 1788744973.844612 punch from 0x4911b4 to 0x4930d6 (0x1f23 bytes) 000340[0] 1788744973.906566 trunc from 0x77e0e6 to 0x40cf4f (0x371196 bytes) 000343[0] 1788744973.977242 mapread 0x309017 thru 0x309eed (0xed7 bytes) 000344[0] 1788744974.062928 mapwrite 0x133348 thru 0x133dab (0xa64 bytes) 000345[1] 1788744974.141326 write 0x12ec15 thru 0x1305f4 (0x19e0 bytes) 000346[1] 1788744974.196477 read 0x198bb5 thru 0x199d83 (0x11cf bytes) 000347[1] 1788744974.239467 mapwrite 0x8becf9 thru 0x8c0574 (0x187c bytes) 000348[0] 1788744974.452683 read 0x684ed3 thru 0x685ae2 (0xc10 bytes) 000349[0] 1788744974.501415 mapread 0x7cae34 thru 0x7cca21 (0x1bee bytes) 000350[0] 1788744974.594400 mapwrite 0x9c3acd thru 0x9c5360 (0x1894 bytes) 000351[0] 1788744974.778382 mapwrite 0x31b930 thru 0x31bdd9 (0x4aa bytes) 000352[1] 1788744974.861504 mapread 0x7d638b thru 0x7d7841 (0x14b7 bytes) 000353[1] 1788744974.935488 trunc from 0x9c5361 to 0x251899 (0x773ac7 bytes) 000354[1] 1788744974.987493 trunc from 0x25189a to 0x2b57f5 (0x63f5c bytes) 000356[1] 1788744975.027696 read 0x13be19 thru 0x13d5e3 (0x17cb bytes) 000357[0] 1788744975.106135 read 0x0c21ab thru 0x0c3b02 (0x1958 bytes) 000358[1] 1788744975.163746 mapread 0x0cbd7e thru 0x0cc8d9 (0xb5c bytes) 000359[0] 1788744975.208740 write 0x9f5e18 thru 0x9f7790 (0x1979 bytes) 000363[0] 1788744975.288759 mapwrite 0x88b43a thru 0x88d1b4 (0x1d7b bytes) 000367[1] 1788744975.441310 mapread 0x198805 thru 0x198d85 (0x581 bytes) 000371[0] 1788744975.548047 mapread 0x6abd4a thru 0x6ac782 (0xa39 bytes) 000372[1] 1788744975.621967 read 0x3c13d4 thru 0x3c1aea (0x717 bytes) 000374[0] 1788744975.623263 punch from 0x5027ad to 0x503d9f (0x15f3 bytes) 000380[0] 1788744975.745597 mapwrite 0x5f5612 thru 0x5f6380 (0xd6f bytes) 000381[1] 1788744975.794483 write 0x5a9a49 thru 0x5ab2e3 (0x189b bytes) 000382[0] 1788744975.820647 mapwrite 0x11172c thru 0x111ea0 (0x775 bytes) 000386[1] 1788744975.960686 write 0x8b4e58 thru 0x8b5bb0 (0xd59 bytes) 000388[0] 1788744976.007653 read 0x161883 thru 0x1629cb (0x1149 bytes) skipping zero size read 000391[1] 1788744976.049770 mapwrite 0x386eba thru 0x386ee4 (0x2b bytes) 000392[0] 1788744976.122563 mapread 0x26344c thru 0x264b6e (0x1723 bytes) 000394[1] 1788744976.172251 trunc from 0x9f7791 to 0x1d1bd0 (0x825bc0 bytes) 000395[0] 1788744976.238469 read 0x129e3f thru 0x12aa31 (0xbf3 bytes) 000396[1] 1788744976.305631 punch from 0x04e754 to 0x04f922 (0x11cf bytes) 000399[0] 1788744976.374981 read 0x122c61 thru 0x122c73 (0x13 bytes) 000400[1] 1788744976.444152 mapread 0x08e09c thru 0x08fa25 (0x198a bytes) 000401[0] 1788744976.458331 read 0x13b517 thru 0x13c932 (0x141c bytes) 000404[1] 1788744976.489590 read 0x126162 thru 0x127d30 (0x1bcf bytes) 000409[0] 1788744976.578360 write 0x15a6be thru 0x15c34a (0x1c8d bytes) 000410[1] 1788744976.654278 mapwrite 0x863197 thru 0x86330f (0x179 bytes) 000411[0] 1788744976.835835 mapwrite 0x7f72ed thru 0x7f8e16 (0x1b2a bytes) 000413[1] 1788744976.957279 mapwrite 0x65e398 thru 0x65e717 (0x380 bytes) 000414[0] 1788744977.120001 trunc from 0x863310 to 0x6a1414 (0x1c1efb bytes) 000416[0] 1788744977.223869 trunc from 0x6a1415 to 0x994f5b (0x2f3b47 bytes) 000417[1] 1788744977.254671 punch from 0x5c500b to 0x5c6252 (0x1248 bytes) 000418[1] 1788744977.320577 read 0x30b1c1 thru 0x30b7bf (0x5ff bytes) 000420[1] 1788744977.347058 write 0x82fe52 thru 0x83016a (0x319 bytes) 000421[1] 1788744977.390226 trunc from 0x994f5c to 0x77bff6 (0x218f65 bytes) 000422[0] 1788744977.439929 punch from 0x54ee79 to 0x54fcfa (0xe82 bytes) 000423[1] 1788744977.476697 punch from 0x749e2d to 0x74acda (0xeae bytes) 000424[0] 1788744977.526105 punch from 0x25062c to 0x2512a9 (0xc7e bytes) 000426[1] 1788744977.584315 mapwrite 0x38f843 thru 0x38fead (0x66b bytes) 000427[1] 1788744977.654699 punch from 0x119a3e to 0x119d90 (0x353 bytes) 000428[0] 1788744977.659286 mapwrite 0x0b0e44 thru 0x0b18c1 (0xa7e bytes) 000429[0] 1788744977.814870 read 0x1f0f3f thru 0x1f2386 (0x1448 bytes) 000433[1] 1788744977.877352 punch from 0x5b7571 to 0x5b87d2 (0x1262 bytes) 000434[0] 1788744977.894636 mapwrite 0x9f345d thru 0x9f45a7 (0x114b bytes) 000436[1] 1788744978.000282 write 0x26cf49 thru 0x26df78 (0x1030 bytes) 000438[1] 1788744978.060372 mapwrite 0x7bdb4c thru 0x7bdf7f (0x434 bytes) 000439[1] 1788744978.151346 trunc from 0x9f45a8 to 0x7c9233 (0x22b374 bytes) 000441[1] 1788744978.222496 read 0x7864e9 thru 0x788009 (0x1b21 bytes) 000442[0] 1788744978.234868 trunc from 0x7c9234 to 0x78e7a3 (0x3aa90 bytes) 000445[0] 1788744978.309197 write 0x72fca3 thru 0x731bc4 (0x1f22 bytes) 000446[0] 1788744978.344289 read 0x6b527d thru 0x6b5726 (0x4aa bytes) 000447[0] 1788744978.349119 trunc from 0x78e7a4 to 0x3751b6 (0x4195ed bytes) 000449[0] 1788744978.404064 read 0x12fb0b thru 0x1305ad (0xaa3 bytes) 000450[1] 1788744978.489817 mapwrite 0x0c46cb thru 0x0c55a6 (0xedc bytes) 000452[0] 1788744978.599874 mapwrite 0x5fe3e0 thru 0x5feb2c (0x74d bytes) 000453[1] 1788744978.762476 write 0x088e6e thru 0x08abd4 (0x1d67 bytes) 000457[0] 1788744978.895601 mapwrite 0x7c4237 thru 0x7c5afe (0x18c8 bytes) 000458[1] 1788744979.136204 mapread 0x643b8b thru 0x64576e (0x1be4 bytes) 000459[1] 1788744979.244082 read 0x7ac068 thru 0x7ac637 (0x5d0 bytes) 000460[1] 1788744979.252221 mapread 0x2ad71c thru 0x2ae35f (0xc44 bytes) 000461[1] 1788744979.298390 mapread 0x7581fc thru 0x758547 (0x34c bytes) 000462[0] 1788744979.308198 write 0x8d9604 thru 0x8d9911 (0x30e bytes) 000463[1] 1788744979.321122 trunc from 0x8d9912 to 0x70fb35 (0x1c9ddc bytes) 000464[1] 1788744979.397075 mapwrite 0x9cfa2a thru 0x9d0e8b (0x1462 bytes) 000465[1] 1788744979.551643 mapread 0x1b2e7e thru 0x1b41e1 (0x1364 bytes) 000466[1] 1788744979.619835 read 0x95428a thru 0x955ae1 (0x1858 bytes) 000467[1] 1788744979.655482 read 0x3d78ae thru 0x3d79ed (0x140 bytes) 000468[0] 1788744979.694001 mapread 0x8924cc thru 0x8932ff (0xe34 bytes) 000471[0] 1788744979.795238 read 0x08da64 thru 0x08e227 (0x7c4 bytes) 000472[0] 1788744979.825716 mapread 0x444025 thru 0x44532e (0x130a bytes) 000474[1] 1788744979.871347 mapread 0x1692ee thru 0x16a930 (0x1643 bytes) 000475[1] 1788744979.880922 read 0x47fdab thru 0x48062c (0x882 bytes) 000476[0] 1788744979.885648 trunc from 0x9d0e8c to 0x0d3cf5 (0x8fd196 bytes) 000477[1] 1788744979.990225 punch from 0x06a8cd to 0x06c89f (0x1fd3 bytes) 000478[1] 1788744980.039504 mapwrite 0x8e48dd thru 0x8e50bd (0x7e1 bytes) 000481[0] 1788744980.231559 mapwrite 0x93d6d0 thru 0x93f5de (0x1f0f bytes) 000482[1] 1788744980.414306 mapread 0x8073d9 thru 0x808383 (0xfab bytes) 000483[0] 1788744980.491296 read 0x364384 thru 0x364702 (0x37f bytes) 000486[1] 1788744980.564493 read 0x48f647 thru 0x4911f3 (0x1bad bytes) 000487[0] 1788744980.591764 punch from 0x16caab to 0x16d476 (0x9cc bytes) 000488[1] 1788744980.631183 trunc from 0x93f5df to 0x583ef8 (0x3bb6e6 bytes) 000489[0] 1788744980.726845 mapwrite 0x9b5910 thru 0x9b6f31 (0x1622 bytes) 000491[0] 1788744980.886426 mapwrite 0x8a9733 thru 0x8a976a (0x38 bytes) 000493[0] 1788744980.977208 mapwrite 0x33b09e thru 0x33b9e5 (0x948 bytes) 000494[1] 1788744981.054361 punch from 0x4966ff to 0x49778f (0x1091 bytes) 000495[0] 1788744981.082742 mapwrite 0x800d15 thru 0x8010c7 (0x3b3 bytes) 000498[1] 1788744981.175408 mapread 0x64408b thru 0x645b01 (0x1a77 bytes) 000500[1] 1788744981.222453 read 0x47ab5a thru 0x47b754 (0xbfb bytes) All operations completed A-OK! PASS 16b (30s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 21:36:26 (1788744986) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 21:36:31 (1788744991) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.339721 s, 24.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.30291 s, 27.7 MB/s ldlm.namespaces.MGC192.168.201.147@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9aa4c2ebe000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9aa4c8847000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c8847000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c8847000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.101 s, 8.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.84433 s, 36.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.64832 s, 39.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.61301 s, 15.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.06573 s, 13.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.08768 s, 25.7 MB/s PASS 16d (50s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 21:37:21 (1788745041) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.297515 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095655 s, 11.0 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 21:37:29 (1788745049) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 21:37:57 (1788745077) TIMEOUT PASS 16g (27s) == sanityn test 16h: mmap read after truncate file ======= 21:38:25 (1788745105) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 21:38:32 (1788745112) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 21:38:39 (1788745119) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.14797 s, 22.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.38332 s, 19.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.72664 s, 9.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.86776 s, 14.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.49709 s, 10.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.33037 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.08141 s, 8.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.7199 s, 15.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.13714 s, 12.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.55483 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.70489 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.74397 s, 9.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.93658 s, 8.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.16348 s, 8.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.40419 s, 7.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.84553 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.09554 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.87491 s, 6.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.87689 s, 6.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (39s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 21:39:18 (1788745158) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Sun Sep 6 21:40:10 EDT 2026 + fsx 2 done: rc=0 Sun Sep 6 21:40:10 EDT 2026 + fsx 3 done: rc=0 Sun Sep 6 21:40:10 EDT 2026 + fsx 4 done: rc=0 Sun Sep 6 21:40:10 EDT 2026 + fsx 5 done: rc=0 Sun Sep 6 21:40:10 EDT 2026 PASS 16k (61s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 21:40:19 (1788745219) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 21:40:30 (1788745230) mmap test1: basic mmap operation (PASS, 0.19878s) mmap test2: MAP_PRIVATE not write back (PASS, 0.082937s) mmap test3: concurrent mmap ops on two nodes (PASS, 14.578s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7772s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.29691s) mmap test6: check mmap write/read content on two nodes (PASS, 0.42041s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 2.0532s) mmap test11: multi-thread shared mmap (PASS, 1.8872s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1776s) PASS 18 (36s) == sanityn test 19: test concurrent uncached read races ========================================================================= 21:41:06 (1788745266) oleg147-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 21:41:12 (1788745272) PASS 20 (9s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 21:41:20 (1788745280) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 21:41:27 (1788745287) now is 1788745290 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 new atime is 1788745351 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 21:42:38 (1788745358) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 21:42:46 (1788745366) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 21:42:53 (1788745373) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 21:42:59 (1788745379) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 21:43:02 (1788745382) utime: good mknod times 1788745383 <= 1788745384 <= 1788745384 for /mnt/lustre/f26a utime: good mknod times 1788745383 <= 1788745384 <= 1788745384 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 21:43:09 (1788745389) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 21:43:18 (1788745398) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-06 21:43:20.000000000 -0400 Birth: 2026-09-06 21:43:20.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-06 21:43:20.000000000 -0400 Birth: 2026-09-06 21:43:20.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 21:43:28 (1788745408) 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 21:43:37 (1788745417) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 21:43:45 (1788745425) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.147@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (23s) == sanityn test 31r: open-rename(replace) race =========== 21:44:08 (1788745448) -rw-r--r-- 1 root root 0 Sep 6 21:44 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 21:44:17 (1788745457) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43186 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 21:44:25 (1788745465) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 21:44:32 (1788745472) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 21:44:36 (1788745476) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.72454 s, 30.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41853 s, 37.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.75639 s, 29.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.92054 s, 27.3 MB/s contention_seconds 4, contention_events 20 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.99258 s, 26.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.23585 s, 23.4 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.13251 s, 46.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.1553 s, 45.4 MB/s PASS 32c (27s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 21:45:05 (1788745505) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 21:45:08 (1788745508) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 21:45:12 (1788745512) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 21:45:15 (1788745515) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 21:45:18 (1788745518) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg147-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9aa4c2ebe000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aa4c2ebe000.ost_server_uuid in IDLE state after 0 sec oleg147-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aa4c2ebe000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9aa4c2ebe000.ost_server_uuid in IDLE state after 0 sec oleg147-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9aa4c2ebe000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aa4c2ebe000.ost_server_uuid in IDLE state after 0 sec oleg147-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aa4c2ebe000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9aa4c2ebe000.ost_server_uuid in IDLE state after 0 sec oleg147-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9aa4c2ebe000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aa4c2ebe000.ost_server_uuid in IDLE state after 0 sec oleg147-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9aa4c2ebe000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9aa4c2ebe000.ost_server_uuid in IDLE state after 0 sec PASS 34 (145s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 21:47:42 (1788745662) fail_loc=0x80000317 Race attempt 0 Wait for 50497 50509 for 60 sec... - open/close 925 (time 1788745682.60 total 10.00 last 92.45) - open/close 1694 (time 1788745692.60 total 20.01 last 76.84) - open/close 2554 (time 1788745702.61 total 30.02 last 85.93) - open/close 3451 (time 1788745712.62 total 40.03 last 89.64) total: 4000 open/close in 46.89 seconds: 85.31 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.147@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.201.147@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (78s) == sanityn test 36: handle ESTALE/open-unlink correctly == 21:49:00 (1788745740) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.64971 s, 31.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.93337 s, 27.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7490560) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.77643 s, 29.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.31114 s, 40.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.52604 s, 34.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.44484 s, 36.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.24138 s, 42.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.10913 s, 47.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37671 s, 38.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(8) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41092 s, 37.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7493632) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.14011 s, 46.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7490560) after_dd(7442432) after(7493632) PASS 36 (288s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 21:53:48 (1788746028) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 - create 1539 (time 1788746042.09 total 10.00 last 153.85) - create 2864 (time 1788746052.09 total 20.00 last 132.50) - create 4409 (time 1788746062.09 total 30.01 last 154.46) - create 5856 (time 1788746072.10 total 40.01 last 144.66) - create 7491 (time 1788746082.10 total 50.01 last 163.40) - create 8949 (time 1788746092.11 total 60.02 last 145.72) total: 10000 create in 67.26 seconds: 148.67 ops/second PASS 37 (80s) == sanityn test 39a: file mtime does not change after rename ========================================================== 21:55:08 (1788746108) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 21:55:15 (1788746115) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 21:55:22 (1788746122) repeat after cancel_lru_locks PASS 39c (10s) == sanityn test 39d: sync write should update mtime ====== 21:55:32 (1788746132) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 21:55:42 (1788746142) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 21:56:00 (1788746160) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 21:56:18 (1788746178) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 21:56:36 (1788746196) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 21:56:53 (1788746213) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 21:57:07 (1788746227) open: File exists Conflict PASS 41b (15s) == sanityn test 41c: pdirops: create vs link ======================================================================== 21:57:22 (1788746242) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (13s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 21:57:35 (1788746255) Conflict PASS 41d (16s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 21:57:51 (1788746271) Conflict PASS 41e (16s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 21:58:07 (1788746287) Conflict PASS 41f (17s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 21:58:24 (1788746304) Conflict PASS 41g (21s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 21:58:45 (1788746325) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (19s) == sanityn test 41i: reint_open: create vs create ======== 21:59:04 (1788746344) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1091s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 22:17:15 (1788747435) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (15s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 22:17:30 (1788747450) open: File exists Conflict PASS 42b (16s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 22:17:46 (1788747466) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 22:18:01 (1788747481) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 22:18:15 (1788747495) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 22:18:28 (1788747508) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 22:18:42 (1788747522) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 22:18:56 (1788747536) Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 22:19:11 (1788747551) PASS 43a (68s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 22:20:19 (1788747619) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 22:20:33 (1788747633) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 22:20:47 (1788747647) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 22:21:01 (1788747661) Conflict PASS 43e (15s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 22:21:17 (1788747677) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 22:21:31 (1788747691) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 22:21:45 (1788747705) Conflict PASS 43h (12s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 22:21:57 (1788747717) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 22:22:01 (1788747721) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (141s) == sanityn test 43k: unlink vs create ==================== 22:24:22 (1788747862) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1169s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 22:43:51 (1788749031) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun Sep 6 22:43:57 EDT 2026 PASS 44a (10s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 22:44:01 (1788749041) open: File exists Conflict PASS 44b (10s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 22:44:11 (1788749051) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (11s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 22:44:22 (1788749062) Conflict PASS 44d (10s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 22:44:32 (1788749072) Conflict PASS 44e (9s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 22:44:42 (1788749082) Conflict PASS 44f (9s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 22:44:51 (1788749091) Conflict PASS 44g (9s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 22:45:00 (1788749100) Conflict PASS 44h (10s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 22:45:10 (1788749110) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 22:45:12 (1788749112) PASS 45a (90s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 22:46:42 (1788749202) Conflict PASS 45b (10s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 22:46:52 (1788749212) Conflict PASS 45c (10s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 22:47:02 (1788749222) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (9s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 22:47:11 (1788749231) Conflict PASS 45e (10s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 22:47:21 (1788749241) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (8s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 22:47:29 (1788749249) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (11s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 22:47:40 (1788749260) Conflict PASS 45h (9s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 22:47:50 (1788749270) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 22:47:52 (1788749272) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1022s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 23:04:54 (1788750294) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (14s) == sanityn test 46b: pdirops: link vs create ======================================================================== 23:05:08 (1788750308) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 23:05:21 (1788750321) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (14s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 23:05:35 (1788750335) Conflict PASS 46d (14s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 23:05:49 (1788750349) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 23:06:02 (1788750362) Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 23:06:17 (1788750377) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 23:06:30 (1788750390) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 23:06:45 (1788750405) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 23:06:48 (1788750408) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 23:06:52 (1788750412) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 23:06:55 (1788750415) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 23:07:00 (1788750420) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 23:07:03 (1788750423) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 23:07:07 (1788750427) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 23:07:11 (1788750431) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 23:07:14 (1788750434) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.116536 s, 87.9 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 23:07:27 (1788750447) 0+1 records in 0+1 records out 158 bytes copied, 0.0181875 s, 8.7 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218429) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 23:07:37 (1788750457) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0140832 s, 72.7 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879081 s, 11.9 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 23:07:58 (1788750478) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0476783 s, 21.5 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 23:08:10 (1788750490) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623928 s, 16.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 23:08:19 (1788750499) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 23:08:27 (1788750507) llite.lustre-ffff9aa4c2ebe000.xattr_cache=1 llite.lustre-ffff9aa4c8847000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:08:29.000000000 -0400 Modify: 2026-09-06 23:08:29.000000000 -0400 Change: 2026-09-06 23:08:29.000000000 -0400 Birth: 2026-09-06 23:08:29.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:08:35.000000000 -0400 Modify: 2026-09-06 23:08:36.000000000 -0400 Change: 2026-09-06 23:08:36.000000000 -0400 Birth: 2026-09-06 23:08:36.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:08:42.000000000 -0400 Modify: 2026-09-06 23:08:42.000000000 -0400 Change: 2026-09-06 23:08:42.000000000 -0400 Birth: 2026-09-06 23:08:42.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:08:42.000000000 -0400 Modify: 2026-09-06 23:08:43.000000000 -0400 Change: 2026-09-06 23:08:43.000000000 -0400 Birth: 2026-09-06 23:08:42.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:08:48.000000000 -0400 Modify: 2026-09-06 23:08:49.000000000 -0400 Change: 2026-09-06 23:08:49.000000000 -0400 Birth: 2026-09-06 23:08:49.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:08:48.000000000 -0400 Modify: 2026-09-06 23:08:49.000000000 -0400 Change: 2026-09-06 23:08:49.000000000 -0400 Birth: 2026-09-06 23:08:49.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (33s) == sanityn test 55a: rename vs unlink target dir ========= 23:09:00 (1788750540) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 23:09:13 (1788750553) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 23:09:27 (1788750567) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 23:09:47 (1788750587) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 23:10:03 (1788750603) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 23:10:07 (1788750607) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 23:10:28 (1788750628) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 23:10:50 (1788750650) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788750652 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.622848 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (106s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 23:12:37 (1788750757) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788750762 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.00061 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (137s) == sanityn test 60: Verify data_version behaviour ======== 23:14:54 (1788750894) PASS 60 (9s) == sanityn test 70a: cd directory && rm directory ======== 23:15:04 (1788750904) PASS 70a (8s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 23:15:12 (1788750912) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 23:15:21 (1788750921) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0230178 s, 2.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 23:15:25 (1788750925) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0261488 s, 2.5 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 23:15:29 (1788750929) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 23:15:32 (1788750932) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0517489 s, 1.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 23:15:37 (1788750937) llite.lustre-ffff9aa4c2ebe000.xattr_cache=1 llite.lustre-ffff9aa4c8847000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 23:15:44 (1788750944) llite.lustre-ffff9aa4c2ebe000.xattr_cache=1 llite.lustre-ffff9aa4c8847000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 23:15:51 (1788750951) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0192929 s, 53.1 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0221767 s, 46.2 kB/s 231236: get lock2 231237: get lock1 231236: done 231236: get lock1 231237: done 231237: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231237: cannot get lock2: Resource deadlock avoided 231237: exit rc=0 231236: done 231236: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 23:16:03 (1788750963) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0963201 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657118 s, 16.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231819 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 23:16:16 (1788750976) opening files: ........ closing files: ........ PASS 76 (77s) == sanityn test 77a: check FIFO NRS policy =============== 23:17:33 (1788751053) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.625943 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702075 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444046 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147764 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151773 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177117 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258809 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305758 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231291 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271926 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35117 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350811 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380852 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318971 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323637 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343938 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.517457 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043429 s, 24.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847729 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262772 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029073 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353573 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965301 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389838 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718922 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10659 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627338 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533411 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548432 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061296 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565122 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615304 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669138 s, 15.7 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 23:17:43 (1788751063) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.309136 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824461 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628805 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181335 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125583 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261371 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233501 s, 4.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269151 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27266 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303731 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307701 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343267 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144298 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00794265 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599769 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324049 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322069 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325144 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00352322 s, 298 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00811307 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723833 s, 145 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355084 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00326517 s, 321 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00484734 s, 216 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233994 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906049 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393645 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120242 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100022 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176376 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00643857 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500851 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109551 s, 9.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=8 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=5 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.389732 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393345 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560405 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648661 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111909 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717343 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101703 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119821 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192838 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106573 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151086 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240983 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198986 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324573 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332992 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341195 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355761 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442439 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477619 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512991 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149612 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839019 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00337602 s, 311 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067112 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076267 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00294512 s, 356 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826749 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148749 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845022 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590656 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251969 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178719 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130915 s, 80.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (12s) == sanityn test 77c: check ORR NRS policy ================ 23:17:56 (1788751076) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.475422 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884453 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120551 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294256 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267266 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277487 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246794 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417235 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412965 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326674 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474197 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476903 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420599 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468883 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.519831 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476672 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.559261 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00794934 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236286 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=8 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00973525 s, 108 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121256 s, 86.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012746 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131703 s, 79.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424951 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00860739 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137552 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19669 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00695967 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134769 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464422 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326476 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00962177 s, 109 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.536106 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387765 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813128 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797936 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158879 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198297 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218076 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297828 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347045 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342688 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305979 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360426 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433837 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422524 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45763 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496301 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.571021 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136266 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310786 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137328 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137597 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183647 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136636 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113994 s, 92.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00230685 s, 455 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00745408 s, 141 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00843335 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00257605 s, 407 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19991 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198761 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124824 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120356 s, 8.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149742 s, 70.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.455824 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762453 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986837 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577944 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13752 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618302 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173253 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083268 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25552 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122331 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246065 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208832 s, 5.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232885 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199674 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143182 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208705 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227863 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383165 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427948 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062219 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351316 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133551 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524982 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477759 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715499 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523184 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694713 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633433 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101001 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769386 s, 13.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943048 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198557 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382861 s, 274 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 23:18:12 (1788751092) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.455635 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622265 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865362 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946608 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124014 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192013 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191559 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220274 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21613 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316548 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282548 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296504 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345886 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296191 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3105 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359923 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400172 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0054072 s, 194 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111302 s, 94.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00558879 s, 188 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00631033 s, 166 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00692556 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104125 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100074 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015834 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571113 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0051032 s, 205 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587267 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106307 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124991 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00904109 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216901 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329086 s, 31.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.725579 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110286 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192033 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183035 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147144 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189548 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109305 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194878 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319574 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114234 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22445 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617004 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299042 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296317 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276006 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39202 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29818 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114264 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104805 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044175 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477705 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529524 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156101 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142845 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656133 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131507 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197729 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804087 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151972 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880414 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133674 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103523 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109248 s, 9.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.527 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116031 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132032 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136127 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130333 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213375 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12494 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266066 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294903 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275614 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287584 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315398 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236705 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373258 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335072 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36332 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295473 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307994 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439233 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727001 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035684 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368082 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418426 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445326 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496229 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515008 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520302 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594529 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060186 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647603 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654296 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675701 s, 15.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742154 s, 14.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 23:18:30 (1788751110) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.47@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.62014 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699127 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534903 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751895 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143905 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453634 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177171 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107784 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137899 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2239 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142536 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189986 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780593 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489689 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344644 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30748 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403781 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551799 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303249 s, 34.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645847 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727472 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768126 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800794 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742645 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940445 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732319 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110053 s, 9.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127797 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113119 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112054 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10669 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126349 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191214 s, 5.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.473828 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689478 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855884 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117804 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215083 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23761 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24061 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257119 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173809 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300708 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242871 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364983 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373843 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390505 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338321 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447256 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066255 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00882636 s, 119 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136591 s, 76.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901883 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253934 s, 413 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143051 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620451 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11487 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00794641 s, 132 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13447 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144684 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104795 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037822 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492363 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347327 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946832 s, 11.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.450417 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738162 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803133 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820758 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219771 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235386 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229226 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191909 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246731 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30904 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334757 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324348 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322651 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278453 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286539 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320839 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33726 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134959 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151321 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142259 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142787 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168139 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987779 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637957 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125726 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127214 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953345 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1508 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105348 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126817 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110095 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120916 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137334 s, 7.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.537574 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828629 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142766 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16794 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890484 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1577 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124387 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17867 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170594 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160862 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792447 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201811 s, 5.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273937 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403446 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352214 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318642 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.515543 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585672 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824081 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334614 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00525609 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00586266 s, 179 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255565 s, 410 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00502355 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828516 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00239528 s, 438 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052886 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044521 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108492 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119706 s, 87.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477821 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801995 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263821 s, 39.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (26s) == sanityn test 77f: check TBF JobID nrs policy ========== 23:18:56 (1788751136) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.39369 s, 42.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573501 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824422 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112095 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812739 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071861 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078193 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13052 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196379 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166763 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162336 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330266 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357283 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29649 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292136 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407893 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457603 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613047 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131023 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429417 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07217 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775125 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869022 s, 12.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862706 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834324 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912359 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941993 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934996 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101191 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10162 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112955 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179004 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187321 s, 5.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.560547 s, 29.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705742 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089353 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737175 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771795 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212448 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231559 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185635 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286354 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209493 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228455 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210623 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298567 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199177 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165316 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308346 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258703 s, 4.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146669 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00281241 s, 373 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143143 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874712 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779686 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16604 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118253 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554677 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457202 s, 22.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861141 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403637 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515941 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037806 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316679 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252386 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152311 s, 68.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.482357 s, 34.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823675 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802665 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873218 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808713 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163082 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465134 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173025 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245843 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131153 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179885 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226751 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305792 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373931 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328587 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394494 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4621 s, 2.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485289 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485022 s, 21.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104921 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052598 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567289 s, 18.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629003 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630376 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592089 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783869 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779934 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820211 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900507 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923644 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925757 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894004 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109648 s, 9.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.46064 s, 36.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089292 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855795 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109264 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129702 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129115 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19552 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328887 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336905 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280979 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25927 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353762 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326906 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334042 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309917 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387085 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412097 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106349 s, 98.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0094107 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566488 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193923 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199619 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00238378 s, 440 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685347 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160455 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00217564 s, 482 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082068 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032717 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037755 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026107 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209872 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148936 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184722 s, 56.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (24s) == sanityn test 77g: Change TBF type directly ============ 23:19:20 (1788751160) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (12s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 23:19:32 (1788751172) oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg147-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg147-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg147-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg147-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 PASS 77h (10s) == sanityn test 77i: Change rank of TBF rule ============= 23:19:42 (1788751182) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (24s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 23:20:06 (1788751206) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.524189 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908775 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101171 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990288 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872713 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190918 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217724 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26467 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349972 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342459 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325076 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308263 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297044 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335983 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28965 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364742 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329524 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373019 s, 28.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488071 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110445 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529603 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999772 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556073 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558721 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552261 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376813 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453624 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561933 s, 18.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556245 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641474 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504494 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488841 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604008 s, 17.4 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.49288 s, 12.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6326 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (61s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 23:21:08 (1788751268) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg147-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg147-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg147-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg147-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31142 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922051 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204312 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205745 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205813 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241455 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267068 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227668 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266104 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257712 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255312 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259876 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296503 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309228 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308229 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325712 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324794 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738186 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863533 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784231 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108854 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653215 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935539 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972534 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107196 s, 9.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140861 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13964 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108959 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115912 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153977 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119692 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1169 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129898 s, 8.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.942 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0128 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg147-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg147-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg147-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg147-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31536 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147903 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131731 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153197 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139397 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154301 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228107 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250703 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240283 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279448 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255222 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186318 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177323 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326651 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391584 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398296 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421291 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323711 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505478 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512801 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461234 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571815 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577594 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653014 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493203 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714689 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580175 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471108 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569841 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080378 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059525 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080162 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681444 s, 15.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9471 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9471 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (150s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 23:23:37 (1788751417) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg147-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg147-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg147-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg147-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27753 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108961 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138064 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147351 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139701 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171423 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189459 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183501 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177836 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201908 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183906 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190031 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191828 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224122 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195021 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974398 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282173 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231895 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834785 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892962 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449938 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883181 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892814 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916917 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107204 s, 9.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842351 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116857 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118766 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123191 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898008 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120692 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114879 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12397 s, 8.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9087 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9544 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg147-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg147-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg147-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg147-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27949 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827727 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752427 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11419 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142805 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119646 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1479 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138922 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140145 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108625 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945656 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136768 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120195 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253318 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248811 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176404 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298609 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832889 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901445 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716828 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896684 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866466 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654957 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756449 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816229 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718098 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784126 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724846 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715695 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701417 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085525 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761985 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974563 s, 10.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9815 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9487 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (148s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 23:26:06 (1788751566) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.4789 s, 35.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588647 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974468 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868286 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110027 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178683 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131488 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186059 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209019 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148727 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223898 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236816 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167197 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173393 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237208 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199366 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238363 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4c2ebe000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4c2ebe000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644478 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597423 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00941885 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00527565 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00698832 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00675996 s, 155 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00787845 s, 133 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00611425 s, 171 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00482483 s, 217 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169559 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977703 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562176 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057555 s, 18.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719725 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536133 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102094 s, 10.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.70213 s, 15.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91075 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.47@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.47@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.445072 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487514 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310465 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058308 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368245 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110294 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137368 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941852 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148145 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145474 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114253 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181964 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184856 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133795 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196144 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210733 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158861 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00580196 s, 181 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021983 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759298 s, 138 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00767066 s, 137 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712727 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040355 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950668 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594853 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389382 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348124 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00656666 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036708 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275825 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301067 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232081 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194058 s, 54.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.97576 s, 15.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86429 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.47@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.379607 s, 44.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483297 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585334 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866211 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133852 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120054 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133078 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198733 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210437 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196209 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180655 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188657 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18358 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214934 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191832 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227522 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241869 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03101 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631794 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415921 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421447 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265455 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445313 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625117 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651558 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492296 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566579 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580548 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065558 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076156 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069745 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714774 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812767 s, 12.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.25958 s, 16.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.28605 s, 19.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.776792 s, 21.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578165 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753635 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154456 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187055 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1192 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193515 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171793 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194869 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165669 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160306 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243286 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153383 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23472 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263917 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167763 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735869 s, 14.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468164 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472817 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501082 s, 20.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447138 s, 23.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04638 s, 22.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481188 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461013 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512228 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056479 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795284 s, 13.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129193 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891762 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876124 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892622 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953884 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108987 s, 9.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8756 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9734 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.60225 s, 15.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0141 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.911611 s, 18.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772905 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894576 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118627 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164438 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164974 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146699 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202032 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191069 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169164 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24275 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203542 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218079 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232048 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221995 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296106 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290928 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658023 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306173 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408559 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660105 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946939 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942105 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839434 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513842 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113056 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123479 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576113 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935185 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936115 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724855 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883916 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964912 s, 10.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8266 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6368 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.446869 s, 37.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062986 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730151 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881835 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839095 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140406 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207064 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222074 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160641 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147396 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218781 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130131 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167504 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252279 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162973 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196473 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202958 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307604 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340753 s, 30.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064807 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368178 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389616 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412608 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422464 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440644 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486375 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497398 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495455 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552028 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506747 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848373 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617344 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615062 s, 17.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.55352 s, 16.0 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95743 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.724457 s, 23.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752969 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815141 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609198 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124462 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111919 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117523 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148193 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227244 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177385 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205417 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273107 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283911 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226936 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292598 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337379 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325113 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338006 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0047034 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623242 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109471 s, 95.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273178 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150713 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00792307 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129061 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00832642 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00857845 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00248105 s, 423 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600232 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981476 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527997 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127722 s, 82.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377194 s, 27.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1556 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9568 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6458 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9107 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (406s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 23:32:52 (1788751972) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.497179 s, 33.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096133 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151812 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916615 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875293 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16871 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158176 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170039 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918436 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188158 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24708 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239625 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26346 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180653 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227734 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320599 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238167 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493061 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606149 s, 17.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982942 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05924 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524984 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730522 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046417 s, 22.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899927 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110132 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849104 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953294 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751554 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770465 s, 13.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861675 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755172 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863845 s, 12.1 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.37161 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90879 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (49s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 23:33:41 (1788752021) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.47@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.47@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.475047 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068245 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109848 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584532 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755538 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121929 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104653 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209588 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147997 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205971 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218515 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21877 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243877 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171278 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11943 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263402 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238027 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669388 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100178 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239261 s, 43.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104215 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420252 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417375 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451724 s, 23.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114008 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494441 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524826 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533238 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528365 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558071 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577853 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621792 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669954 s, 15.7 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.35983 s, 16.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90452 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (46s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 23:34:27 (1788752067) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.47@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.576237 s, 29.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976376 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942175 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182597 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152756 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217434 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234244 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203599 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269742 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270262 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28761 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248189 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280801 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252485 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26753 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340949 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328054 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799902 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473234 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044827 s, 23.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046573 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504778 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513493 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696998 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555703 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580145 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684796 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557222 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862959 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066589 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614376 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700249 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715964 s, 14.6 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.003 s, 15.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12355 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (42s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 23:35:09 (1788752109) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.584261 s, 28.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888883 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612486 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126566 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634059 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147014 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100094 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143541 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119502 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958026 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131338 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276474 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273191 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480529 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475978 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397066 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424214 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211343 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616469 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034205 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673196 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396909 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430816 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437447 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760644 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786358 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598772 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066514 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659992 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901438 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746143 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120683 s, 8.7 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5095 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85983 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.64301 s, 15.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87136 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (90s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 23:36:39 (1788752199) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.04366 s, 16.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108148 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103268 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183788 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189757 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178415 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193723 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128789 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123128 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254319 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147852 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234686 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257764 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147282 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270086 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209106 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972534 s, 10.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424212 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470481 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494475 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510754 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644506 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051146 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671757 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541486 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641614 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588137 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105188 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822388 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641211 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764461 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831624 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104912 s, 10.0 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9044 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9605 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (73s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 23:37:52 (1788752272) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.443377 s, 37.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561171 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575973 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133673 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12472 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152302 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187286 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196798 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197083 s, 5.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190299 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185735 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211414 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198781 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196682 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196649 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216141 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218353 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605097 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309692 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615533 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659912 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671087 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695479 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723968 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746623 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745452 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753372 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802275 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105676 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796281 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626684 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623929 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771848 s, 13.6 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.02008 s, 17.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.83216 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg147-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.537334 s, 31.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567014 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640695 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110749 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139264 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204644 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207883 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196863 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213254 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220333 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19549 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23536 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180199 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21825 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223494 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220261 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184427 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400296 s, 26.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170944 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377632 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296141 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218137 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381951 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400382 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475427 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386646 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902617 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534042 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400375 s, 262 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442988 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425615 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705845 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100498 s, 10.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1569 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82802 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99064 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92911 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (129s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 23:40:01 (1788752401) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg147-client.virtnet /mnt/lustre (opts:) Stopping client oleg147-client.virtnet /mnt/lustre opts: Stopping clients: oleg147-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg147-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg147-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg147-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg147-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.47,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg147-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.47,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg147-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.47,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69719/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.47,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg147-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing load_modules_local oleg147-server: kmemleak disabled oleg147-server: Loading modules from /home/green/git/lustre-release/lustre oleg147-server: detected 4 online CPUs by sysfs oleg147-server: MODOPTS_LIBCFS= oleg147-server: Force libcfs to create 2 CPU partitions oleg147-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre Starting client oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre Started clients oleg147-client.virtnet: 192.168.201.147@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 Starting client oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 Started clients oleg147-client.virtnet: 192.168.201.147@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9aa4c617d000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9aa4e038a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa4c617d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa4e038a000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.472067 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605799 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632009 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445921 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135526 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100609 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13256 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169504 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158393 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174005 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163486 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186111 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200669 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159604 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206722 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187238 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163145 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017703 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244117 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246535 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052111 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740549 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461789 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467833 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00319838 s, 328 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338807 s, 30.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499366 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513552 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050733 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523664 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795343 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127929 s, 82.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820182 s, 12.8 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.43714 s, 16.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17384 s, 20.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.363406 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735041 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630695 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114326 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181876 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20938 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2445 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256775 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16366 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212956 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189753 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256521 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174537 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215388 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187609 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288195 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227432 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538666 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215553 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439178 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499043 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681146 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446295 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529255 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684785 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719991 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759912 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105836 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746648 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057406 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577431 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663513 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807578 s, 13.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.04777 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94855 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (177s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 23:42:58 (1788752578) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg147-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (23s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 23:43:21 (1788752601) On MGS 192.168.201.147, active = nodemap.active=1 On MGS 192.168.201.147, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.823475 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122686 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118176 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2043 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120425 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267487 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291723 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276388 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097925 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335006 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240808 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458402 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46647 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368291 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327011 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52001 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4e038a000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4e038a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456313 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135412 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852914 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136586 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116851 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646048 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149454 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135485 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112678 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139221 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103976 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166712 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108648 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10805 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102694 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112473 s, 9.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4e038a000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146691 s, 7.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.17805 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.39123 s, 19.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.201.147, TBF.id = On MGS 192.168.201.147, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.414339 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054488 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389924 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059566 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132991 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159492 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188471 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196569 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171185 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230114 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203933 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180998 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236439 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227325 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233995 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20874 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312678 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228681 s, 459 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024393 s, 430 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582752 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372633 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378327 s, 277 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4e038a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513209 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665756 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469508 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680239 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918221 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095963 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066345 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693328 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1266 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698425 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030991 s, 33.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.38676 s, 12.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.57627 s, 40.7 MB/s Read runtime is 3 s, speed is 33.333333 IOPS read rate (33.333333) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg147-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg147-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 On MGS 192.168.201.147, active = nodemap.active=1 On MGS 192.168.201.147, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.636381 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704136 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755969 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689452 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19523 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214516 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216989 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266709 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282332 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277649 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285764 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300504 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288727 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295304 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337374 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329778 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345147 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9aa4e038a000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9aa4e038a000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580134 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640725 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906913 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652668 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698594 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732292 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803613 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817216 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788812 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874007 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875891 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143127 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895499 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985223 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980544 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103641 s, 10.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.96609 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0807 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.147, TBF.id = On MGS 192.168.201.147, active = nodemap.active=0 PASS 77kj (191s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 23:46:33 (1788752793) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 23:46:44 (1788752804) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg147-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.69035 s, 284 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg147-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0625 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (57s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 23:47:41 (1788752861) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.42134 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662128 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702698 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198352 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18529 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967207 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828862 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257867 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425734 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120414 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188655 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123329 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177234 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26983 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260951 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260279 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017612 s, 59.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9aa4e038a000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301236 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602079 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14497 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270118 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137014 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386245 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314941 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134351 s, 78.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21119 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336162 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323751 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318844 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190822 s, 55.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00331528 s, 316 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392337 s, 2.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg147-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg147-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.9458 s, 15.1 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.12023 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.577962 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689113 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744183 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917629 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146126 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681701 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047107 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986428 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220102 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207793 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173995 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902474 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193262 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797354 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191812 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542209 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481776 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570911 s, 18.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412354 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454842 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045031 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666358 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058925 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499125 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843158 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717435 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644503 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669945 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822236 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816119 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742738 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693649 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813003 s, 12.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg147-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg147-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.66271 s, 13.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.37057 s, 19.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (84s) == sanityn test 77o: Changing rank should not panic ====== 23:49:05 (1788752945) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (11s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 23:49:16 (1788752956) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (105s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 23:51:01 (1788753061) mds.MDS.mdt.nrs_policies=tbf oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg147-client: oleg147-server: ssh exited with exit code 36 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 oleg147-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg147-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg147-client: oleg147-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (33s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 23:51:34 (1788753094) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273330 kill md_thread xargs PASS 77r (52s) == sanityn test 77s: Check TBF LRU shrinker ============== 23:52:26 (1788753146) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (65s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 23:53:33 (1788753213) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (14s) == sanityn test 79: xattr: intent error ================== 23:53:46 (1788753226) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935687 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 23:53:48.000000000 -0400 Modify: 2026-09-06 23:53:48.000000000 -0400 Change: 2026-09-06 23:53:48.000000000 -0400 Birth: 2026-09-06 23:53:49.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 23:54:05 (1788753245) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 23:54:08 (1788753248) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 23:54:11 (1788753251) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 23:54:15 (1788753255) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 23:54:19 (1788753259) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 23:54:22 (1788753262) - open/close 572 (time 1788753277.00 total 10.00 last 57.18) - open/close 1254 (time 1788753287.04 total 20.04 last 67.97) - open/close 1851 (time 1788753297.04 total 30.04 last 59.68) total: 2500 open/close in 38.43 seconds: 65.05 ops/second snapshot_time 1788753390.891972557 secs.nsecs start_time 1788753265.725091199 secs.nsecs elapsed_time 125.166881358 secs.nsecs open 2505 samples [usecs] 132 16484 4175625 13726424441 close 2504 samples [usecs] 167 8050 971821 918909651 mknod 2501 samples [usecs] 506 16373 3594435 10405970161 unlink 2500 samples [usecs] 709 67474 4626748 25976173834 mkdir 2 samples [usecs] 1106 3198 4304 11450440 rmdir 2 samples [usecs] 13814 16504 30318 463208612 rename 2500 samples [usecs] 429 47305 15789043 214564227657 getattr 3763 samples [usecs] 8 16032 227072 386491396 parallel_rename_file 2500 samples [usecs] 429 47305 15789043 214564227657 crossdir_rename 2500 samples [usecs] 429 47305 15789043 214564227657 readpage 12 samples [usecs] 4491 34582 185307 4011273369 PASS 81d (133s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 23:56:35 (1788753395) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7646 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7646 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 23:56:41 (1788753401) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 23:56:45 (1788753405) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 23:56:58 (1788753418) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788753420 Finishing test test1 at 1788753425 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 23:57:10 (1788753430) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 23:57:14 (1788753434) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 23:57:17 (1788753437) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 23:57:21 (1788753441) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5813 0x16b5 0x240000400 1 5773 0x168d 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5774 0x168e 0x280000400 0 5814 0x16b6 0x240000400 rc1=0 and rc2=0 PASS 93 (15s) == sanityn test 94: signal vs CP callback race =========== 23:57:36 (1788753456) osc.lustre-OST0000-osc-ffff9aa4c617d000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9aa4e038a000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9aa4c617d000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9aa4e038a000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.216008 s, 37.9 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00451 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290443 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0936681 s, 43.7 kB/s osc.lustre-OST0000-osc-ffff9aa4c617d000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9aa4e038a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa4c617d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9aa4e038a000.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 23:57:52 (1788753472) llite.lustre-ffff9aa4c617d000.fast_read=0 llite.lustre-ffff9aa4e038a000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.092616 s, 22.6 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9aa4c617d000.fast_read=1 llite.lustre-ffff9aa4e038a000.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 23:58:13 (1788753493) llite.lustre-ffff9aa4c617d000.fast_read=0 llite.lustre-ffff9aa4e038a000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00679371 s, 1.8 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9aa4c617d000.fast_read=1 llite.lustre-ffff9aa4e038a000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 23:58:27 (1788753507) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 23:58:31 (1788753511) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0064772 s, 632 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (9s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 23:58:39 (1788753519) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187753 s, 11.2 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (6s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 23:58:46 (1788753526) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17682 s, 11.9 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 23:58:53 (1788753533) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00682087 s, 1.8 MB/s PASS 100e (6s) == sanityn test 101a: Discard DoM data on unlink ========= 23:58:59 (1788753539) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0113879 s, 360 kB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 23:59:05 (1788753545) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00751097 s, 545 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 23:59:12 (1788753552) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00906586 s, 452 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 23:59:20 (1788753560) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 68 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf68:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (7s) == sanityn test 103: Test size correctness with lockahead ========================================================== 23:59:27 (1788753567) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788753570 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788753573 Finishing test test23 at 1788753573 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 23:59:41 (1788753581) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 23:59:44 (1788753584) fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 00:00:12 (1788753612) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 00:00:15 (1788753615) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0114289 s, 89.6 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 00:00:17.000000000 -0400 Modify: 2026-09-07 00:00:17.000000000 -0400 Change: 2026-09-07 00:00:17.000000000 -0400 Birth: 2026-09-07 00:00:18.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938223 -rw-r--r-- 1 0 0 1788753618 1788753617 1788753617 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 00:00:17.000000000 -0400 Modify: 2026-09-07 00:00:17.000000000 -0400 Change: 2026-09-07 00:00:17.000000000 -0400 Birth: 2026-09-07 00:00:18.000000000 -0400 1788753617 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 00:00:23 (1788753623) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 00:00:30 (1788753630) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.296166 s, 35.4 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5781 0x1695 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 00:00:39 (1788753639) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.236142 s, 44.4 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5823 0x16bf 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 00:00:52 (1788753652) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00346331 s, 2.9 MB/s llite.lustre-ffff9aa4c617d000.hybrid_io=0 llite.lustre-ffff9aa4e038a000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.69991 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 00:01:05 (1788753665) 192.168.201.147@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 192.168.201.147@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing load_modules_local oleg147-server: kmemleak disabled oleg147-server: Loading modules from /home/green/git/lustre-release/lustre oleg147-server: detected 4 online CPUs by sysfs oleg147-server: MODOPTS_LIBCFS= oleg147-server: Force libcfs to create 2 CPU partitions oleg147-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 192.168.201.147@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing load_modules_local oleg147-server: kmemleak disabled oleg147-server: Loading modules from /home/green/git/lustre-release/lustre oleg147-server: detected 4 online CPUs by sysfs oleg147-server: MODOPTS_LIBCFS= oleg147-server: Force libcfs to create 2 CPU partitions oleg147-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 PASS 109 (68s) == sanityn test 110: do not grant another lock on resend ========================================================== 00:02:13 (1788753733) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 00:02:15.000000000 -0400 Modify: 2026-09-07 00:02:15.000000000 -0400 Change: 2026-09-07 00:02:15.000000000 -0400 Birth: 2026-09-07 00:02:16.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg147-client.virtnet oleg147-client.virtnet PASS 110 (168s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 00:05:01 (1788753901) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 00:05:06 (1788753906) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 00:05:09 (1788753909) lustre-OST0000-osc-ffff9aa4e038a800 active. lustre-OST0001-osc-ffff9aa4e038a800 active. lustre-OST0000-osc-ffff9aa4e038a800 active. lustre-OST0001-osc-ffff9aa4e038a800 active. MGC192.168.201.147@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 00:05:16 (1788753916) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 00:05:20 (1788753920) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 00:05:24 (1788753924) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 00:05:28 (1788753928) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0158641 s, 1.3 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.112725 s, 182 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 00:05:37 (1788753937) PASS 122 (10s) == sanityn test 200: service remains healthy while able to process request ========================================================== 00:05:47 (1788753947) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1322 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1804 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1706 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1272 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1992 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2302 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2249 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1761 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3764 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3658 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3589 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3908 s, 0.1 kB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.38 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4365 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3238 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5032 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4588 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5328 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg147-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9aa4c617d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9aa4c617d800.ost_server_uuid in IDLE state after 0 sec PASS 200 (81s) cleanup: ====================================================== == sanityn test complete, duration 9562 sec ============== 00:07:11 (1788754031) === sanityn: start cleanup 00:07:12 (1788754032) === Stopping clients: oleg147-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg147-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 00:11:13 (1788754273) ===