-----============= acceptance-small: sanityn ============----- Wed Sep 16 18:05:14 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg633-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 18:05:36 (1789596336) === Starting client oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 Mount client oleg633-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 Started clients oleg633-client.virtnet: 192.168.206.133@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg633-client.virtnet: executing check_config_client /mnt/lustre oleg633-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg633-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff936444f00800.idle_timeout=debug osc.lustre-OST0000-osc-ffff936448070000.idle_timeout=debug osc.lustre-OST0001-osc-ffff936444f00800.idle_timeout=debug osc.lustre-OST0001-osc-ffff936448070000.idle_timeout=debug disable quota as required oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 18:06:12 (1789596372) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468423 s, 22.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7650] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 18:06:15 (1789596375) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 18:06:22 (1789596382) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (5s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 18:06:27 (1789596387) -rw-r--r-- 1 root root 0 Sep 16 18:06 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 18:06:33 (1789596393) -rw-r--r-- 1 root root 0 Sep 16 18:06 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 18:06:39 (1789596399) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 18:06:46 (1789596406) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 18:06:52 (1789596412) total 1 -rwxrwxrwx 1 root root 0 Sep 16 18:06 f2a -rwxrwxrwx 1 root root 0 Sep 16 18:06 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 16 18:06 f2a -rwxrwxrwx 1 root root 0 Sep 16 18:06 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0275378 s, 18.6 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 18:06:58 (1789596418) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 18:07:02 (1789596422) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.572528 s, 3.7 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 18:07:10 (1789596430) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 18:07:18 (1789596438) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 18:07:26 (1789596446) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 18:07:34 (1789596454) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 18:07:41 (1789596461) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 18:07:48 (1789596468) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 18:07:55 (1789596475) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 18:08:03 (1789596483) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478127 s, 21.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0484378 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 18:08:10 (1789596490) 1+0 records in 1+0 records out 1 byte copied, 0.0147281 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0133335 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102921 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0121653 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0107739 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0151895 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0134374 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0179898 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0246093 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0188601 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0226827 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0305697 s, 0.0 kB/s PASS 10a (9s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 18:08:19 (1789596499) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0211729 s, 145 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0564525 s, 72.6 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000708116 s, 4.3 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 18:08:26 (1789596506) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (9s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 18:08:35 (1789596515) warning: '-runtime' deprecated, use '-t runtime' instead - link 324 (time 1789596527.84 total 10.05 last 32.24) - link 643 (time 1789596537.85 total 20.06 last 31.86) - link 954 (time 1789596547.85 total 30.07 last 31.08) - link 1278 (time 1789596557.87 total 40.08 last 32.36) - link 1603 (time 1789596567.90 total 50.11 last 32.40) - link 1911 (time 1789596577.91 total 60.12 last 30.76) - link 2215 (time 1789596587.95 total 70.16 last 30.28) - link 2505 (time 1789596597.99 total 80.20 last 28.90) - link 2810 (time 1789596608.00 total 90.21 last 30.47) total: 3116 link in 100.01 seconds: 31.16 ops/second using seed 2535306602 running for 100 seconds - stat 5252 (time 1789596529 ; total 11 ; last 11) - stat 10000 (time 1789596537 ; total 19 ; last 8) - stat 14827 (time 1789596548 ; total 30 ; last 11) - stat 20000 (time 1789596557 ; total 39 ; last 9) - stat 24895 (time 1789596568 ; total 50 ; last 11) - stat 27378 (time 1789596579 ; total 61 ; last 11) - stat 29892 (time 1789596590 ; total 72 ; last 11) - stat 30000 (time 1789596590 ; total 72 ; last 0) - stat 32522 (time 1789596601 ; total 83 ; last 11) - stat 35348 (time 1789596612 ; total 94 ; last 11) total: 36685 stats in 100 seconds: 366.850006 stats/second - unlinked 0 (time 1789596568 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3116) error: No such file or directory total: 3115 unlinks in 89 seconds: 35.000000 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21447) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21449) - No such process PASS 12 (148s) == sanityn test 13: test directory page revocation ======= 18:11:03 (1789596663) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 18:11:13 (1789596673) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 22634 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 18:11:20 (1789596680) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 23213 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 18:11:27 (1789596687) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23791 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 18:11:35 (1789596695) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24373 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 18:11:44 (1789596704) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24957 Terminated $DIR1/$tdir/sleep 60 PASS 14d (11s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 18:11:55 (1789596715) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (56s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 18:12:51 (1789596771) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.34869 s, 6.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.921123 s, 9.1 MB/s Chance of close/open is 1 in 50 Seed set to 6778 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1789596780.554226 write 0x1190c9 thru 0x11d080 (0x3fb8 bytes) 000003[0] 1789596780.633971 punch from 0x0310a9 to 0x03a859 (0x97b1 bytes) 000004[1] 1789596780.681565 write 0x08c11c thru 0x0931aa (0x708f bytes) 000005[0] 1789596780.831633 punch from 0x0a2c1b to 0x0a899f (0x5d85 bytes) 000006[1] 1789596780.911947 write 0x31efd0 thru 0x32081a (0x184b bytes) 000007[0] 1789596781.011993 read 0x2370ae thru 0x245b09 (0xea5c bytes) 000009[0] 1789596781.117906 write 0x69e851 thru 0x69f612 (0xdc2 bytes) 000012[1] 1789596781.159349 mapwrite 0x3f0bfb thru 0x3f1fa5 (0x13ab bytes) truncating to largest ever: 0x4fd479 000013[0] 1789596781.430141 trunc from 0x69f613 to 0x4fd478 (0x1a219a bytes) 000018[1] 1789596781.648721 read 0x3c54b0 thru 0x3cc5b4 (0x7105 bytes) 000019[0] 1789596781.724495 punch from 0x4c28b1 to 0x4cdea3 (0xb5f3 bytes) 000021[0] 1789596781.794310 write 0x527488 thru 0x52f379 (0x7ef2 bytes) 000023[0] 1789596781.843362 mapread 0x2024c7 thru 0x2099d4 (0x750e bytes) 000025[1] 1789596781.873901 mapread 0x22aa3f thru 0x23a1dc (0xf79e bytes) 000026[0] 1789596781.945209 mapread 0x05ec5a thru 0x068be4 (0x9f8b bytes) 000029[1] 1789596782.055361 trunc from 0x52f37a to 0x44fa2c (0xdf94d bytes) 000031[0] 1789596782.133020 punch from 0x2a9fb1 to 0x2b66f0 (0xc740 bytes) 000033[1] 1789596782.179706 punch from 0x26f089 to 0x274cfd (0x5c75 bytes) 000034[1] 1789596782.197957 punch from 0x033461 to 0x039eaf (0x6a4f bytes) 000036[0] 1789596782.275772 mapwrite 0x435713 thru 0x4441f4 (0xeae2 bytes) 000038[1] 1789596782.454981 write 0x7f043e thru 0x7f2bcd (0x2790 bytes) 000039[0] 1789596782.526312 read 0x1fd805 thru 0x20654c (0x8d48 bytes) 000042[1] 1789596782.599498 mapread 0x31cc18 thru 0x32061b (0x3a04 bytes) 000047[1] 1789596782.696895 mapwrite 0x1ff053 thru 0x20345a (0x4408 bytes) 000048[1] 1789596782.927550 mapwrite 0x378d4b thru 0x37cf86 (0x423c bytes) 000050[1] 1789596783.028570 mapread 0x1bab01 thru 0x1c951e (0xea1e bytes) 000053[0] 1789596783.082272 read 0x4dcaf6 thru 0x4ec937 (0xfe42 bytes) 000054[1] 1789596783.192967 punch from 0x4f0ea3 to 0x4fe710 (0xd86e bytes) 000055[1] 1789596783.225164 read 0x6cebb9 thru 0x6d15d5 (0x2a1d bytes) 000056[0] 1789596783.233242 write 0x1ebab0 thru 0x1ee264 (0x27b5 bytes) 000058[0] 1789596783.367146 write 0x8a7fe3 thru 0x8aa81f (0x283d bytes) 000060[1] 1789596783.401821 mapread 0x444900 thru 0x448f1d (0x461e bytes) 000061[1] 1789596783.461918 trunc from 0x8aa820 to 0x436c69 (0x473bb6 bytes) 000062[1] 1789596783.506825 punch from 0x28789d to 0x28829e (0xa02 bytes) 000064[0] 1789596783.556805 mapwrite 0x596405 thru 0x5a59c6 (0xf5c2 bytes) truncating to largest ever: 0x6b5322 000066[0] 1789596783.915806 trunc from 0x5a59c7 to 0x6b5321 (0x10f95b bytes) 000067[1] 1789596783.993806 mapwrite 0x734752 thru 0x73a27d (0x5b2c bytes) 000070[1] 1789596784.311165 read 0x35885c thru 0x36817f (0xf924 bytes) 000071[0] 1789596784.351863 mapread 0x2815e1 thru 0x28dff2 (0xca12 bytes) 000072[1] 1789596784.430887 read 0x24b6f7 thru 0x24ec53 (0x355d bytes) 000074[1] 1789596784.455194 mapwrite 0x24476a thru 0x24bcf5 (0x758c bytes) truncating to largest ever: 0x8b106e 000076[1] 1789596784.574016 trunc from 0x73a27e to 0x8b106d (0x176df0 bytes) 000078[0] 1789596784.619722 mapread 0x1a5f3f thru 0x1ad09b (0x715d bytes) 000080[0] 1789596784.662945 mapwrite 0x0600e0 thru 0x063f5b (0x3e7c bytes) 000081[0] 1789596784.831382 mapread 0x62d2c1 thru 0x63704a (0x9d8a bytes) 000082[0] 1789596784.874419 write 0x45a685 thru 0x469d0e (0xf68a bytes) 000083[1] 1789596784.925531 write 0x9474a7 thru 0x956117 (0xec71 bytes) 000086[0] 1789596785.001468 mapwrite 0x5a006f thru 0x5a810e (0x80a0 bytes) 000087[1] 1789596785.220885 punch from 0x93ca23 to 0x94bcd6 (0xf2b4 bytes) 000088[0] 1789596785.271824 mapwrite 0x99c891 thru 0x9a1daf (0x551f bytes) 000089[0] 1789596785.389790 read 0x1ad4f9 thru 0x1adccd (0x7d5 bytes) 000091[1] 1789596785.395848 trunc from 0x9a1db0 to 0x43aabf (0x5672f0 bytes) 000093[1] 1789596785.442595 mapwrite 0x54ba6c thru 0x55561d (0x9bb2 bytes) 000094[0] 1789596785.579507 write 0x82a90b thru 0x8364bf (0xbbb5 bytes) 000095[0] 1789596785.614954 trunc from 0x8364c0 to 0x18818b (0x6ae334 bytes) 000096[1] 1789596785.675382 trunc from 0x18818c to 0x6026e9 (0x47a55e bytes) 000098[0] 1789596785.773805 read 0x3d1c6e thru 0x3d30c7 (0x145a bytes) 000099[1] 1789596785.803664 trunc from 0x6026ea to 0x6d4c6e (0xd2585 bytes) 000100[0] 1789596785.860416 write 0x58fd43 thru 0x59c658 (0xc916 bytes) 000101[0] 1789596785.896835 punch from 0x11627d to 0x120fb6 (0xad3a bytes) 000102[0] 1789596785.990335 mapwrite 0x8d693b thru 0x8d9c90 (0x3356 bytes) 000103[0] 1789596786.228750 punch from 0x67d9b6 to 0x68bf22 (0xe56d bytes) 000104[1] 1789596786.248847 read 0x3cbc8c thru 0x3d2a45 (0x6dba bytes) 000105[0] 1789596786.286759 mapread 0x0af59d thru 0x0b07a8 (0x120c bytes) 000107[0] 1789596786.377810 mapwrite 0x7ec478 thru 0x7f29a9 (0x6532 bytes) 000108[0] 1789596786.631040 mapwrite 0x248131 thru 0x24a532 (0x2402 bytes) 000109[1] 1789596786.781381 mapwrite 0x4a30e7 thru 0x4a756a (0x4484 bytes) 000110[1] 1789596787.071808 write 0x1e9217 thru 0x1f0aea (0x78d4 bytes) 000111[0] 1789596787.112997 mapread 0x1fc3a6 thru 0x202e8a (0x6ae5 bytes) 000112[0] 1789596787.208983 mapread 0x7d866f thru 0x7daa1a (0x23ac bytes) 000113[0] 1789596787.276745 write 0x739ed2 thru 0x73cffc (0x312b bytes) 000114[0] 1789596787.364818 read 0x335162 thru 0x3423c8 (0xd267 bytes) 000115[0] 1789596787.432603 punch from 0x4a693c to 0x4ad05a (0x671f bytes) 000116[0] 1789596787.439862 read 0x8c116d thru 0x8c2da1 (0x1c35 bytes) 000117[0] 1789596787.448949 write 0x0e1ffe thru 0x0e8849 (0x684c bytes) 000120[1] 1789596787.560213 punch from 0x0c8507 to 0x0cad3a (0x2834 bytes) 000122[0] 1789596787.627067 write 0x990b20 thru 0x994124 (0x3605 bytes) 000124[1] 1789596787.695048 read 0x891625 thru 0x899a2e (0x840a bytes) 000125[1] 1789596787.768497 trunc from 0x994125 to 0x181f3d (0x8121e7 bytes) 000126[0] 1789596787.846310 write 0x1810ac thru 0x1867fe (0x5753 bytes) 000127[1] 1789596787.907474 mapread 0x01e564 thru 0x021548 (0x2fe5 bytes) 000129[1] 1789596787.988383 write 0x93c7f3 thru 0x93d975 (0x1183 bytes) truncating to largest ever: 0x9b1432 000130[1] 1789596788.019821 trunc from 0x93d976 to 0x9b1431 (0x73abc bytes) 000133[0] 1789596788.071606 punch from 0x258fd6 to 0x261d4d (0x8d78 bytes) 000135[1] 1789596788.138362 mapread 0x5c6e21 thru 0x5cfa7c (0x8c5c bytes) 000136[1] 1789596788.242705 write 0x91a4ee thru 0x91d7e9 (0x32fc bytes) 000137[0] 1789596788.267303 write 0x409a93 thru 0x40ff6e (0x64dc bytes) 000138[0] 1789596788.324429 mapwrite 0x41301c thru 0x418e4b (0x5e30 bytes) 000139[1] 1789596788.431494 mapread 0x60f5bd thru 0x61ce0f (0xd853 bytes) 000142[0] 1789596788.502947 write 0x0bfbf8 thru 0x0cdb10 (0xdf19 bytes) 000143[0] 1789596788.540826 write 0x04d3fb thru 0x05c311 (0xef17 bytes) 000144[1] 1789596788.558859 read 0x4f3a7a thru 0x4fb872 (0x7df9 bytes) 000145[0] 1789596788.587961 read 0x16d560 thru 0x173014 (0x5ab5 bytes) 000147[0] 1789596788.629243 trunc from 0x9b1432 to 0x698feb (0x318446 bytes) truncating to largest ever: 0x9d63a0 000148[0] 1789596788.664647 trunc from 0x698fec to 0x9d639f (0x33d3b4 bytes) 000149[0] 1789596788.742619 write 0x928a23 thru 0x92abe6 (0x21c4 bytes) 000150[1] 1789596788.802570 read 0x6582f5 thru 0x65fcf7 (0x7a03 bytes) 000151[1] 1789596788.865659 write 0x68848c thru 0x68dc25 (0x579a bytes) 000152[0] 1789596788.940188 write 0x757b5d thru 0x757ca8 (0x14c bytes) 000153[0] 1789596788.998867 mapwrite 0x80b9d3 thru 0x817cd9 (0xc307 bytes) 000154[0] 1789596789.138928 mapwrite 0x01d152 thru 0x026f37 (0x9de6 bytes) 000155[0] 1789596789.266335 trunc from 0x9d63a0 to 0x86ffc7 (0x1663d8 bytes) 000157[0] 1789596789.449131 punch from 0x2faa9e to 0x2ff8fd (0x4e60 bytes) 000158[1] 1789596789.476982 mapread 0x755992 thru 0x76341b (0xda8a bytes) 000161[0] 1789596789.653907 trunc from 0x86ffc8 to 0x0267c7 (0x849800 bytes) 000164[0] 1789596789.800551 mapread 0x01d119 thru 0x01e579 (0x1461 bytes) 000166[0] 1789596789.832099 punch from 0x01cfb6 to 0x0267c7 (0x9812 bytes) 000169[1] 1789596789.878032 write 0x5fb594 thru 0x603f8a (0x89f7 bytes) 000170[0] 1789596789.916984 trunc from 0x603f8b to 0x749ac8 (0x145b3e bytes) 000171[1] 1789596789.963905 trunc from 0x749ac9 to 0x6d218c (0x7793c bytes) 000174[1] 1789596790.042028 punch from 0x5faca4 to 0x60354d (0x88aa bytes) 000176[0] 1789596790.073309 punch from 0x28594d to 0x293474 (0xdb28 bytes) 000177[0] 1789596790.110439 trunc from 0x6d218d to 0x77cebf (0xaad33 bytes) 000178[0] 1789596790.161884 mapwrite 0x53d5eb thru 0x542bcb (0x55e1 bytes) 000180[0] 1789596790.280350 write 0x86ed74 thru 0x876887 (0x7b14 bytes) 000183[1] 1789596790.321800 write 0x976a9b thru 0x97de88 (0x73ee bytes) 000184[0] 1789596790.373452 mapread 0x42834f thru 0x429d1a (0x19cc bytes) 000185[1] 1789596790.438387 punch from 0x17eeab to 0x186859 (0x79af bytes) 000188[0] 1789596790.586969 read 0x8a339a thru 0x8ade24 (0xaa8b bytes) 000189[0] 1789596790.643339 punch from 0x29e34f to 0x2a2e8c (0x4b3e bytes) 000191[0] 1789596790.680087 punch from 0x07841a to 0x0811a1 (0x8d88 bytes) 000195[1] 1789596790.782114 read 0x85b44f thru 0x85c20c (0xdbe bytes) 000196[0] 1789596790.839296 read 0x9488b2 thru 0x95662c (0xdd7b bytes) 000197[1] 1789596790.896334 trunc from 0x97de89 to 0x91a6db (0x637ad bytes) 000198[0] 1789596790.994704 mapread 0x4add08 thru 0x4ba003 (0xc2fc bytes) 000199[0] 1789596791.096393 read 0x836729 thru 0x8451fd (0xead5 bytes) 000200[0] 1789596791.155018 mapread 0x0f4341 thru 0x100d5d (0xca1d bytes) 000201[0] 1789596791.219873 write 0x112581 thru 0x1180e0 (0x5b60 bytes) 000202[0] 1789596791.283900 read 0x44825d thru 0x453198 (0xaf3c bytes) 000206[1] 1789596791.355287 mapread 0x2728b5 thru 0x27b8cd (0x9019 bytes) 000207[0] 1789596791.445074 trunc from 0x91a6dc to 0x302e5a (0x617881 bytes) 000208[1] 1789596791.571100 mapread 0x2c6607 thru 0x2d0dd4 (0xa7ce bytes) 000209[0] 1789596791.633197 punch from 0x1dedda to 0x1e383a (0x4a61 bytes) 000213[1] 1789596791.708636 mapwrite 0x40a839 thru 0x41255b (0x7d23 bytes) 000214[0] 1789596791.850037 trunc from 0x41255c to 0x94edc0 (0x53c865 bytes) 000215[0] 1789596791.902797 trunc from 0x94edc1 to 0x306eef (0x647ed1 bytes) 000218[1] 1789596791.953760 trunc from 0x306ef0 to 0x6dbd27 (0x3d4e38 bytes) 000220[0] 1789596792.005945 punch from 0x586fb1 to 0x58e973 (0x79c3 bytes) 000221[1] 1789596792.041129 mapwrite 0x507ad6 thru 0x51693c (0xee67 bytes) 000223[0] 1789596792.191604 mapwrite 0x903745 thru 0x9062db (0x2b97 bytes) 000224[0] 1789596792.298813 mapwrite 0x699076 thru 0x69be18 (0x2da3 bytes) 000226[0] 1789596792.420621 mapwrite 0x027ea4 thru 0x028267 (0x3c4 bytes) 000227[1] 1789596792.499036 mapread 0x8792d2 thru 0x87df9a (0x4cc9 bytes) 000228[0] 1789596792.575019 mapread 0x54ecda thru 0x55e0a9 (0xf3d0 bytes) 000229[0] 1789596792.613120 read 0x2a2f9f thru 0x2aa936 (0x7998 bytes) 000230[1] 1789596792.630901 trunc from 0x9062dc to 0x3fdad2 (0x508809 bytes) 000233[1] 1789596792.812329 mapwrite 0x18bbb7 thru 0x1978c5 (0xbd0f bytes) 000235[0] 1789596792.970696 write 0x10f395 thru 0x11b3c7 (0xc033 bytes) 000241[1] 1789596793.168809 trunc from 0x3fdad3 to 0x4fab0c (0xfd03a bytes) 000242[1] 1789596793.255075 trunc from 0x4fab0d to 0x7beb95 (0x2c4089 bytes) 000243[0] 1789596793.276102 write 0x6b4ad1 thru 0x6c1153 (0xc683 bytes) 000245[1] 1789596793.330269 write 0x194265 thru 0x199d0e (0x5aaa bytes) 000246[0] 1789596793.379061 mapread 0x104b72 thru 0x10e3ba (0x9849 bytes) 000247[0] 1789596793.481070 write 0x1af93b thru 0x1bd2f8 (0xd9be bytes) 000248[0] 1789596793.515421 trunc from 0x7beb96 to 0x812948 (0x53db3 bytes) 000249[1] 1789596793.544731 write 0x67cda2 thru 0x684d24 (0x7f83 bytes) 000250[1] 1789596793.597067 trunc from 0x812949 to 0x2ce39d (0x5445ab bytes) 000251[1] 1789596793.651480 read 0x186c7f thru 0x1893ad (0x272f bytes) 000253[0] 1789596793.721623 mapwrite 0x9a8ab1 thru 0x9b1d68 (0x92b8 bytes) 000254[0] 1789596793.996323 mapread 0x590b33 thru 0x5926d1 (0x1b9f bytes) 000257[0] 1789596794.071623 mapread 0x78a0d9 thru 0x78ff49 (0x5e71 bytes) 000258[0] 1789596794.130167 punch from 0x323adb to 0x32e429 (0xa94f bytes) 000260[0] 1789596794.162378 write 0x33f8c6 thru 0x34e675 (0xedb0 bytes) 000262[0] 1789596794.195010 read 0x470e77 thru 0x4803ab (0xf535 bytes) 000267[0] 1789596794.306762 read 0x0769ce thru 0x07a7b0 (0x3de3 bytes) 000268[0] 1789596794.315789 mapwrite 0x98d436 thru 0x99a0c7 (0xcc92 bytes) 000270[0] 1789596794.414922 mapread 0x23cff8 thru 0x240a15 (0x3a1e bytes) 000271[0] 1789596794.462689 mapread 0x7a1660 thru 0x7a8ecc (0x786d bytes) 000273[0] 1789596794.550446 read 0x4947c5 thru 0x49db79 (0x93b5 bytes) 000274[0] 1789596794.566695 mapwrite 0x0baf13 thru 0x0c528a (0xa378 bytes) 000276[0] 1789596794.667582 mapread 0x83f7fb thru 0x841227 (0x1a2d bytes) 000277[0] 1789596794.693072 read 0x67ca1b thru 0x67dfac (0x1592 bytes) 000279[1] 1789596794.782008 mapread 0x8fe1ae thru 0x90bd97 (0xdbea bytes) 000282[0] 1789596794.873047 punch from 0x9411bb to 0x942b9f (0x19e5 bytes) 000284[1] 1789596794.889990 mapread 0x6d968a thru 0x6df06d (0x59e4 bytes) 000287[0] 1789596795.001349 punch from 0x021f96 to 0x028370 (0x63db bytes) 000288[0] 1789596795.007133 read 0x4584bd thru 0x45b3a7 (0x2eeb bytes) 000289[1] 1789596795.048451 mapread 0x59cc1f thru 0x5a8791 (0xbb73 bytes) 000291[1] 1789596795.123987 punch from 0x772d7d to 0x77b6cf (0x8953 bytes) 000293[1] 1789596795.176805 read 0x570cdb thru 0x57bc04 (0xaf2a bytes) 000294[1] 1789596795.208609 trunc from 0x9b1d69 to 0x48cf4c (0x524e1c bytes) 000295[1] 1789596795.266326 mapwrite 0x56cb1d thru 0x577427 (0xa90b bytes) 000296[0] 1789596795.510206 punch from 0x4754f4 to 0x4786db (0x31e8 bytes) 000301[0] 1789596795.751133 punch from 0x0b777a to 0x0b9d22 (0x25a9 bytes) 000303[1] 1789596795.824119 mapread 0x503249 thru 0x507f0d (0x4cc5 bytes) 000305[0] 1789596795.944506 mapread 0x002a23 thru 0x011d65 (0xf343 bytes) 000308[0] 1789596796.021885 write 0x1fbc08 thru 0x1fedf6 (0x31ef bytes) 000311[1] 1789596796.057774 read 0x08c508 thru 0x08d39d (0xe96 bytes) 000312[1] 1789596796.081151 trunc from 0x577428 to 0x4825b4 (0xf4e73 bytes) 000313[0] 1789596796.120408 mapwrite 0x8c980c thru 0x8d6805 (0xcffa bytes) 000314[0] 1789596796.377101 trunc from 0x8d6806 to 0x594fa5 (0x341860 bytes) 000316[1] 1789596796.402077 write 0x052944 thru 0x05de17 (0xb4d4 bytes) 000317[0] 1789596796.432948 trunc from 0x594fa6 to 0x45466e (0x140937 bytes) 000318[1] 1789596796.480886 trunc from 0x45466f to 0x0910fc (0x3c3572 bytes) 000319[1] 1789596796.529293 trunc from 0x0910fd to 0x76b18b (0x6da08f bytes) 000320[0] 1789596796.546013 punch from 0x411579 to 0x41ce22 (0xb8aa bytes) 000322[0] 1789596796.607857 write 0x586331 thru 0x58ebb8 (0x8888 bytes) 000323[1] 1789596796.636225 trunc from 0x76b18c to 0x4473a9 (0x323de2 bytes) 000325[0] 1789596796.705386 read 0x2b9c15 thru 0x2bcaac (0x2e98 bytes) 000328[1] 1789596796.768971 mapread 0x3dd780 thru 0x3e7cdb (0xa55c bytes) 000329[0] 1789596796.831466 mapwrite 0x5c2e7c thru 0x5cb2f9 (0x847e bytes) 000331[0] 1789596797.192753 write 0x2d2b9c thru 0x2e1d19 (0xf17e bytes) 000332[1] 1789596797.254605 trunc from 0x5cb2fa to 0x7a3fc4 (0x1d8ccb bytes) 000333[0] 1789596797.337961 write 0x0ed6d1 thru 0x0f7735 (0xa065 bytes) 000334[0] 1789596797.381616 mapread 0x094a2b thru 0x0a3a64 (0xf03a bytes) 000335[1] 1789596797.444895 mapwrite 0x12868e thru 0x12a0b1 (0x1a24 bytes) 000336[0] 1789596797.614229 trunc from 0x7a3fc5 to 0x9c400a (0x220046 bytes) 000339[0] 1789596797.699499 trunc from 0x9c400b to 0x2b07c0 (0x71384a bytes) 000340[0] 1789596797.759903 mapwrite 0x86e381 thru 0x870612 (0x2292 bytes) 000342[1] 1789596798.041034 mapread 0x2d8193 thru 0x2de010 (0x5e7e bytes) 000343[1] 1789596798.138498 mapwrite 0x1132a7 thru 0x1219ab (0xe705 bytes) 000344[0] 1789596798.466142 trunc from 0x870613 to 0x2310f4 (0x63f51e bytes) 000349[0] 1789596798.678580 punch from 0x0506ec to 0x053902 (0x3217 bytes) 000350[0] 1789596798.736383 mapwrite 0x880a34 thru 0x883bf6 (0x31c3 bytes) 000356[1] 1789596799.081443 mapread 0x42c63f thru 0x43ab21 (0xe4e3 bytes) 000357[0] 1789596799.148410 punch from 0x87f328 to 0x883bf6 (0x48cf bytes) 000358[0] 1789596799.160940 trunc from 0x883bf7 to 0x96c7f2 (0xe8bfc bytes) truncating to largest ever: 0x9e7d72 000359[1] 1789596799.210597 trunc from 0x96c7f3 to 0x9e7d71 (0x7b57f bytes) 000360[1] 1789596799.301780 trunc from 0x9e7d72 to 0x74980b (0x29e566 bytes) 000366[1] 1789596799.444930 read 0x0cbd62 thru 0x0d3a8e (0x7d2d bytes) 000368[1] 1789596799.525756 mapwrite 0x0d5c47 thru 0x0e3a73 (0xde2d bytes) 000370[0] 1789596799.717303 write 0x3ec53e thru 0x3f53b7 (0x8e7a bytes) 000371[0] 1789596799.769839 write 0x9a24b9 thru 0x9a4f65 (0x2aad bytes) 000372[1] 1789596799.786890 punch from 0x1e7958 to 0x1ee6f4 (0x6d9d bytes) 000374[1] 1789596799.876649 mapread 0x5c3806 thru 0x5ca6f3 (0x6eee bytes) 000375[1] 1789596799.923392 write 0x0b9774 thru 0x0c5a9e (0xc32b bytes) 000377[1] 1789596799.988204 punch from 0x0fb84d to 0x10860e (0xcdc2 bytes) 000378[1] 1789596800.064419 read 0x664ed6 thru 0x6731a6 (0xe2d1 bytes) 000379[0] 1789596800.085752 trunc from 0x9a4f66 to 0x50c458 (0x498b0d bytes) 000380[0] 1789596800.177902 mapread 0x0658be thru 0x06e7e4 (0x8f27 bytes) 000381[0] 1789596800.225747 punch from 0x25f51e to 0x26e841 (0xf324 bytes) 000382[0] 1789596800.252981 mapread 0x0ad1bb thru 0x0aecfa (0x1b40 bytes) 000383[0] 1789596800.282461 trunc from 0x50c459 to 0x65875c (0x14c304 bytes) 000385[0] 1789596800.345828 mapwrite 0x5d8bd3 thru 0x5e895e (0xfd8c bytes) 000387[0] 1789596800.456746 mapwrite 0x0e1a68 thru 0x0efd68 (0xe301 bytes) 000388[1] 1789596800.570142 trunc from 0x65875d to 0x952f7f (0x2fa823 bytes) 000389[0] 1789596800.652015 write 0x7eb1f1 thru 0x7f7e1f (0xcc2f bytes) 000390[0] 1789596800.691391 write 0x7ac7fe thru 0x7b0be1 (0x43e4 bytes) 000393[1] 1789596800.749806 mapread 0x6aa853 thru 0x6b2ffc (0x87aa bytes) 000394[1] 1789596800.817102 punch from 0x589857 to 0x5955fe (0xbda8 bytes) 000398[0] 1789596800.885175 write 0x8d53b4 thru 0x8dd4d7 (0x8124 bytes) 000399[1] 1789596800.921315 mapread 0x5d0b0b thru 0x5d54e3 (0x49d9 bytes) 000401[1] 1789596800.971807 write 0x53f73f thru 0x547b76 (0x8438 bytes) 000402[1] 1789596801.007348 write 0x9bc3cf thru 0x9c9284 (0xceb6 bytes) 000405[1] 1789596801.074185 read 0x5f2a4e thru 0x5f54a1 (0x2a54 bytes) 000406[1] 1789596801.080849 read 0x03d2d1 thru 0x041197 (0x3ec7 bytes) 000407[1] 1789596801.089446 read 0x0147c7 thru 0x01ea67 (0xa2a1 bytes) 000408[1] 1789596801.114453 mapwrite 0x55cc02 thru 0x56be85 (0xf284 bytes) 000410[0] 1789596801.211715 write 0x806424 thru 0x815cce (0xf8ab bytes) 000411[1] 1789596801.275008 mapread 0x43cc28 thru 0x43f01a (0x23f3 bytes) 000414[1] 1789596801.359444 mapread 0x2f53f5 thru 0x3040e1 (0xeced bytes) 000417[0] 1789596801.525330 write 0x96d96a thru 0x97d969 (0x10000 bytes) 000418[0] 1789596801.582503 read 0x89dd8b thru 0x8ac61a (0xe890 bytes) 000421[0] 1789596801.676128 punch from 0x3366e2 to 0x33a54f (0x3e6e bytes) 000422[0] 1789596801.727008 mapwrite 0x1c316b thru 0x1d2e7d (0xfd13 bytes) 000429[0] 1789596802.149228 mapwrite 0x509b1a thru 0x511071 (0x7558 bytes) 000431[1] 1789596802.408360 mapwrite 0x2fd07f thru 0x300d0f (0x3c91 bytes) 000432[0] 1789596802.599037 read 0x41dcac thru 0x421585 (0x38da bytes) 000434[0] 1789596802.686574 read 0x7af5b5 thru 0x7bcf07 (0xd953 bytes) 000435[0] 1789596802.743213 trunc from 0x9c9285 to 0x47ba45 (0x54d83f bytes) 000437[0] 1789596802.862740 mapread 0x31fd45 thru 0x3275b3 (0x786f bytes) 000438[1] 1789596802.909701 write 0x0d18d2 thru 0x0e0983 (0xf0b2 bytes) 000439[0] 1789596802.987292 read 0x4480d5 thru 0x456b6a (0xea96 bytes) 000441[1] 1789596803.029201 trunc from 0x47ba46 to 0x6b9675 (0x23dc30 bytes) 000442[0] 1789596803.112831 punch from 0x2ee3ba to 0x2f5a51 (0x7698 bytes) 000444[1] 1789596803.181716 read 0x5d2eb9 thru 0x5d965d (0x67a5 bytes) 000445[1] 1789596803.214354 read 0x10cf56 thru 0x11264f (0x56fa bytes) 000446[1] 1789596803.261025 trunc from 0x6b9676 to 0x965f1c (0x2ac8a7 bytes) 000449[1] 1789596803.411063 mapwrite 0x4b8fd7 thru 0x4c35d1 (0xa5fb bytes) 000450[1] 1789596803.497950 write 0x72a918 thru 0x72accb (0x3b4 bytes) 000451[1] 1789596803.514015 mapread 0x1757c3 thru 0x17df73 (0x87b1 bytes) 000454[1] 1789596803.645931 mapwrite 0x3ab57b thru 0x3b5915 (0xa39b bytes) 000456[1] 1789596803.909118 mapwrite 0x5b72a1 thru 0x5b9d24 (0x2a84 bytes) 000459[1] 1789596804.140637 punch from 0x6aa52d to 0x6b759e (0xd072 bytes) 000460[1] 1789596804.183431 trunc from 0x965f1d to 0x7a9f2d (0x1bbfef bytes) 000461[0] 1789596804.205329 trunc from 0x7a9f2e to 0x25b35f (0x54ebce bytes) 000466[1] 1789596804.385411 write 0x250731 thru 0x2585f2 (0x7ec2 bytes) 000467[0] 1789596804.441862 mapread 0x002426 thru 0x0048f0 (0x24cb bytes) 000468[1] 1789596804.518931 punch from 0x0d1590 to 0x0df3cc (0xde3d bytes) 000469[1] 1789596804.552133 trunc from 0x25b360 to 0x46c886 (0x211527 bytes) 000470[1] 1789596804.594823 write 0x1ab9d2 thru 0x1b27b5 (0x6de4 bytes) 000471[0] 1789596804.642078 read 0x37e1dd thru 0x382369 (0x418d bytes) 000472[1] 1789596804.700645 write 0x7d56ff thru 0x7dfc1a (0xa51c bytes) 000474[1] 1789596804.710461 mapread 0x776ef6 thru 0x77e28b (0x7396 bytes) 000475[0] 1789596804.758605 trunc from 0x7dfc1b to 0x43b508 (0x3a4712 bytes) 000477[0] 1789596804.872057 trunc from 0x43b509 to 0x5cf808 (0x194300 bytes) 000478[1] 1789596804.910501 write 0x42e113 thru 0x43478f (0x667d bytes) 000479[0] 1789596804.962477 mapwrite 0x4d8513 thru 0x4e3b9b (0xb689 bytes) 000481[0] 1789596805.165564 mapwrite 0x3eca9c thru 0x3f012b (0x3690 bytes) 000482[1] 1789596805.222076 trunc from 0x5cf809 to 0x78ff06 (0x1c06fe bytes) 000484[0] 1789596805.270663 read 0x5a97c3 thru 0x5b5d0d (0xc54b bytes) 000486[1] 1789596805.371986 read 0x0e1b97 thru 0x0e8b2b (0x6f95 bytes) 000487[0] 1789596805.426142 punch from 0x0c0abf to 0x0c303e (0x2580 bytes) 000488[0] 1789596805.450013 trunc from 0x78ff07 to 0x461d95 (0x32e171 bytes) 000489[0] 1789596805.555007 read 0x3f464e thru 0x3ffa3f (0xb3f2 bytes) 000490[1] 1789596805.606684 mapread 0x1d3095 thru 0x1e097f (0xd8eb bytes) 000491[0] 1789596805.714358 punch from 0x1fbc1e to 0x2053bf (0x97a2 bytes) 000492[1] 1789596805.766789 write 0x589d36 thru 0x593c4b (0x9f16 bytes) 000493[0] 1789596805.810739 mapwrite 0x682896 thru 0x68b015 (0x8780 bytes) 000494[1] 1789596805.980813 mapread 0x2f4d1b thru 0x304c35 (0xff1b bytes) 000496[0] 1789596806.094411 punch from 0x32136f to 0x3275b9 (0x624b bytes) 000497[1] 1789596806.179782 mapwrite 0x5d9ef3 thru 0x5e281b (0x8929 bytes) 000499[0] 1789596806.444423 read 0x22c395 thru 0x238723 (0xc38f bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 6806 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole 000003[1] 1789596807.329546 write 0x337000 thru 0x33e2c1 (0x72c2 bytes) 000004[1] 1789596807.361033 mapwrite 0x08d000 thru 0x0978ff (0xa900 bytes) truncating to largest ever: 0x8b57ad 000005[1] 1789596807.437235 trunc from 0x33e2c2 to 0x8b57ac (0x5774eb bytes) 000006[0] 1789596807.469703 punch from 0x3df1d8 to 0x3e6f86 (0x7daf bytes) 000007[1] 1789596807.495204 read 0x6d1000 thru 0x6d3441 (0x2442 bytes) 000008[0] 1789596807.524040 read_OD 0x555000 thru 0x560fff (0xc000 bytes) 000009[0] 1789596807.545794 mapwrite 0x1da000 thru 0x1e0453 (0x6454 bytes) 000010[1] 1789596807.785742 punch from 0x8b2293 to 0x8b57ac (0x351a bytes) 000011[1] 1789596807.807550 punch from 0x565e46 to 0x56e283 (0x843e bytes) 000012[1] 1789596807.813194 mapread 0x4c0000 thru 0x4c7c59 (0x7c5a bytes) 000013[0] 1789596807.850307 read_OD 0x109000 thru 0x113fff (0xb000 bytes) 000017[1] 1789596807.882008 mapread 0x1bd000 thru 0x1cba70 (0xea71 bytes) 000018[0] 1789596807.924109 mapread 0x221000 thru 0x22efe0 (0xdfe1 bytes) 000019[1] 1789596807.961041 read 0x80d000 thru 0x814c16 (0x7c17 bytes) 000020[0] 1789596807.973808 write_OD 0x133000 thru 0x138fff (0x6000 bytes) 000021[1] 1789596808.107870 mapread 0x27c000 thru 0x28a2d1 (0xe2d2 bytes) 000027[0] 1789596808.157664 punch from 0x341fc7 to 0x34a3be (0x83f8 bytes) truncating to largest ever: 0x8e7d4e 000029[1] 1789596808.192286 trunc from 0x8b57ad to 0x8e7d4d (0x325a1 bytes) 000031[0] 1789596808.241392 write_OD 0x85a000 thru 0x85bfff (0x2000 bytes) 000032[1] 1789596808.297645 trunc from 0x8e7d4e to 0x1f9643 (0x6ee70a bytes) 000033[1] 1789596808.345257 punch from 0x182adf to 0x192168 (0xf68a bytes) 000034[1] 1789596808.373358 read 0x146000 thru 0x154e14 (0xee15 bytes) 000035[1] 1789596808.463687 mapwrite 0x949000 thru 0x955db2 (0xcdb3 bytes) 000036[0] 1789596808.891579 trunc from 0x955db3 to 0x3ce9c8 (0x5873ea bytes) 000037[0] 1789596808.945239 mapwrite 0x088000 thru 0x092dca (0xadcb bytes) 000040[1] 1789596809.122822 trunc from 0x3ce9c9 to 0x423f58 (0x55590 bytes) 000041[1] 1789596809.172682 write 0x7a9000 thru 0x7aa5dc (0x15dd bytes) 000042[0] 1789596809.175683 read_OD 0x62f000 thru 0x63dfff (0xf000 bytes) 000045[1] 1789596809.232117 read 0x5b8000 thru 0x5bbc8b (0x3c8c bytes) 000046[0] 1789596809.240700 write_OD 0x305000 thru 0x30efff (0xa000 bytes) 000048[1] 1789596809.404410 mapwrite 0x56f000 thru 0x5732af (0x42b0 bytes) 000049[1] 1789596809.639190 read 0x75c000 thru 0x76370d (0x770e bytes) 000050[0] 1789596809.666563 mapread 0x2af000 thru 0x2b3bb9 (0x4bba bytes) 000052[1] 1789596809.846346 mapread 0x132000 thru 0x14103c (0xf03d bytes) 000054[0] 1789596809.916241 read_OD 0x74d000 thru 0x75afff (0xe000 bytes) 000055[1] 1789596809.941795 read 0x635000 thru 0x6418ea (0xc8eb bytes) 000056[1] 1789596809.962826 mapread 0x364000 thru 0x36ebac (0xabad bytes) 000057[1] 1789596810.014847 punch from 0x450043 to 0x45bc61 (0xbc1f bytes) 000061[1] 1789596810.138767 mapwrite 0x604000 thru 0x61032c (0xc32d bytes) 000062[1] 1789596810.299947 mapread 0x317000 thru 0x31f350 (0x8351 bytes) 000063[1] 1789596810.323574 mapread 0x797000 thru 0x7a45e0 (0xd5e1 bytes) 000064[1] 1789596810.368695 mapread 0x2fb000 thru 0x301536 (0x6537 bytes) 000065[0] 1789596810.388486 write_OD 0x51c000 thru 0x526fff (0xb000 bytes) 000067[0] 1789596810.466962 write_OD 0x15e000 thru 0x168fff (0xb000 bytes) 000069[1] 1789596810.562106 write 0x44e000 thru 0x44f178 (0x1179 bytes) 000073[0] 1789596810.629829 mapwrite 0x351000 thru 0x3538fc (0x28fd bytes) 000075[0] 1789596810.759393 mapread 0x52a000 thru 0x52f874 (0x5875 bytes) 000079[0] 1789596810.883966 mapwrite 0x3e6000 thru 0x3f1007 (0xb008 bytes) 000080[0] 1789596811.035792 trunc from 0x7aa5dd to 0x482e48 (0x327794 bytes) 000083[0] 1789596811.151154 punch from 0x0e4cf6 to 0x0ef7fd (0xab08 bytes) 000085[1] 1789596811.156986 mapwrite 0x9ea000 thru 0x9edaba (0x3abb bytes) 000089[1] 1789596811.423254 read 0x2d0000 thru 0x2d03c4 (0x3c5 bytes) 000090[0] 1789596811.428477 read_OD 0x313000 thru 0x313fff (0x1000 bytes) 000091[0] 1789596811.453697 mapread 0x52e000 thru 0x53aa51 (0xca52 bytes) 000093[0] 1789596811.519683 mapwrite 0x944000 thru 0x9520ea (0xe0eb bytes) 000095[1] 1789596811.713897 punch from 0x21eeb7 to 0x224fe9 (0x6133 bytes) 000096[0] 1789596811.739804 read_OD 0x7d0000 thru 0x7defff (0xf000 bytes) 000097[0] 1789596811.772880 mapwrite 0x14a000 thru 0x15529b (0xb29c bytes) 000098[0] 1789596811.905942 trunc from 0x9edabb to 0x56b3c9 (0x4826f1 bytes) 000100[1] 1789596811.928575 mapwrite 0x46e000 thru 0x479d8a (0xbd8b bytes) 000101[0] 1789596812.084657 mapwrite 0x934000 thru 0x93a83c (0x683d bytes) 000104[0] 1789596812.282728 write_OD 0x6ef000 thru 0x6f4fff (0x6000 bytes) 000105[0] 1789596812.394947 punch from 0x29d8d9 to 0x2a472f (0x6e57 bytes) 000112[0] 1789596812.528442 trunc from 0x93a83d to 0x4255d9 (0x515263 bytes) 000114[0] 1789596812.662419 mapread 0x3a2000 thru 0x3a9e84 (0x7e85 bytes) 000115[1] 1789596812.737735 mapread 0x078000 thru 0x07cd40 (0x4d41 bytes) 000117[1] 1789596812.808254 mapread 0x112000 thru 0x11d0df (0xb0e0 bytes) 000118[1] 1789596812.849505 mapread 0x3c2000 thru 0x3cd560 (0xb561 bytes) 000122[0] 1789596812.887359 trunc from 0x4255da to 0x3c67b5 (0x5ee24 bytes) 000125[1] 1789596812.940286 write 0x679000 thru 0x67f3bc (0x63bd bytes) 000129[1] 1789596812.981874 write 0x343000 thru 0x34d2f9 (0xa2fa bytes) 000130[1] 1789596813.016741 mapwrite 0x511000 thru 0x512cf1 (0x1cf2 bytes) 000131[0] 1789596813.205876 read_OD 0x21c000 thru 0x220fff (0x5000 bytes) truncating to largest ever: 0x921757 000132[0] 1789596813.256217 trunc from 0x67f3bd to 0x921756 (0x2a239a bytes) 000133[1] 1789596813.324029 write 0x6eb000 thru 0x6f704e (0xc04f bytes) 000134[0] 1789596813.358068 mapread 0x6e2000 thru 0x6e5f1d (0x3f1e bytes) 000135[0] 1789596813.452401 mapwrite 0x1dd000 thru 0x1e36aa (0x66ab bytes) 000136[0] 1789596813.606314 read_OD 0x4c6000 thru 0x4cffff (0xa000 bytes) 000137[1] 1789596813.641751 write 0x92d000 thru 0x9381fc (0xb1fd bytes) 000141[0] 1789596813.725151 mapwrite 0x480000 thru 0x487075 (0x7076 bytes) 000142[1] 1789596813.971748 punch from 0x722871 to 0x72a73f (0x7ecf bytes) 000145[0] 1789596814.033470 read_OD 0x40b000 thru 0x419fff (0xf000 bytes) 000150[0] 1789596814.097217 punch from 0x1b1893 to 0x1b2e83 (0x15f1 bytes) 000151[0] 1789596814.184565 punch from 0x3de7ca to 0x3ec6ca (0xdf01 bytes) 000152[0] 1789596814.189367 write_OD 0x917000 thru 0x91dfff (0x7000 bytes) 000153[1] 1789596814.305802 mapwrite 0x136000 thru 0x140c5c (0xac5d bytes) 000154[0] 1789596814.428743 read_OD 0x142000 thru 0x14efff (0xd000 bytes) 000155[0] 1789596814.465245 read_OD 0x730000 thru 0x738fff (0x9000 bytes) 000156[1] 1789596814.491047 trunc from 0x9381fd to 0x892dab (0xa5451 bytes) 000158[0] 1789596814.562738 mapread 0x0c5000 thru 0x0cc093 (0x7094 bytes) 000160[0] 1789596814.664446 mapread 0x44b000 thru 0x44e297 (0x3298 bytes) 000161[1] 1789596814.691752 read 0x61a000 thru 0x61d79c (0x379d bytes) 000163[0] 1789596814.706013 punch from 0x5474cb to 0x554def (0xd925 bytes) 000164[0] 1789596814.741258 mapread 0x49d000 thru 0x4a9677 (0xc678 bytes) 000165[1] 1789596814.788369 mapwrite 0x1f9000 thru 0x205b80 (0xcb81 bytes) 000166[0] 1789596815.021912 mapwrite 0x3ba000 thru 0x3bb8f8 (0x18f9 bytes) 000168[1] 1789596815.209060 punch from 0x6ad497 to 0x6ba21e (0xcd88 bytes) 000171[0] 1789596815.244005 trunc from 0x892dac to 0x6ef6f1 (0x1a36ba bytes) 000172[1] 1789596815.291739 write 0x365000 thru 0x3693ef (0x43f0 bytes) 000173[0] 1789596815.319851 mapwrite 0x078000 thru 0x0855c7 (0xd5c8 bytes) 000174[1] 1789596815.598535 write 0x6cd000 thru 0x6d5e63 (0x8e64 bytes) 000175[0] 1789596815.646946 write_OD 0x491000 thru 0x498fff (0x8000 bytes) 000177[1] 1789596815.752735 mapwrite 0x843000 thru 0x849502 (0x6503 bytes) 000179[0] 1789596816.007054 punch from 0x1e25f1 to 0x1efd4e (0xd75e bytes) 000180[1] 1789596816.084100 mapwrite 0x3f5000 thru 0x3fd642 (0x8643 bytes) 000184[0] 1789596816.305021 write_OD 0x3b5000 thru 0x3bffff (0xb000 bytes) 000185[1] 1789596816.356305 mapread 0x55d000 thru 0x56cfe3 (0xffe4 bytes) 000186[1] 1789596816.391480 write 0x266000 thru 0x267b90 (0x1b91 bytes) 000187[0] 1789596816.419693 write_OD 0x11c000 thru 0x11cfff (0x1000 bytes) 000188[1] 1789596816.481311 trunc from 0x849503 to 0x440fae (0x408554 bytes) 000192[0] 1789596816.536665 punch from 0x2accb9 to 0x2b5c4e (0x8f96 bytes) 000193[1] 1789596816.554418 read 0x0f8000 thru 0x0f8e2e (0xe2f bytes) 000194[0] 1789596816.567438 mapread 0x1a2000 thru 0x1a90da (0x70db bytes) 000195[0] 1789596816.622209 punch from 0x27b927 to 0x2897ae (0xde88 bytes) 000197[1] 1789596816.724655 read 0x0e2000 thru 0x0efa74 (0xda75 bytes) 000201[0] 1789596816.790007 trunc from 0x440faf to 0x78ccab (0x34bcfd bytes) 000202[0] 1789596816.908168 trunc from 0x78ccac to 0x1d96c7 (0x5b35e4 bytes) 000203[0] 1789596816.961050 mapwrite 0x4ae000 thru 0x4b5d1d (0x7d1e bytes) 000205[0] 1789596817.212446 read_OD 0x418000 thru 0x41cfff (0x5000 bytes) 000207[1] 1789596817.246481 write 0x218000 thru 0x21dfca (0x5fcb bytes) 000210[1] 1789596817.357862 mapwrite 0x836000 thru 0x83f599 (0x959a bytes) 000212[0] 1789596817.535059 punch from 0x307f79 to 0x30d455 (0x54dd bytes) 000214[1] 1789596817.585969 read 0x6b6000 thru 0x6bd4ee (0x74ef bytes) 000215[1] 1789596817.651749 read 0x07b000 thru 0x07d331 (0x2332 bytes) 000216[1] 1789596817.659192 write 0x14e000 thru 0x14f927 (0x1928 bytes) 000217[0] 1789596817.677712 read_OD 0x478000 thru 0x482fff (0xb000 bytes) 000221[0] 1789596817.762936 mapwrite 0x7a3000 thru 0x7a6af9 (0x3afa bytes) 000222[0] 1789596817.897333 write_OD 0x92d000 thru 0x937fff (0xb000 bytes) 000223[1] 1789596817.961927 punch from 0x5db6ad to 0x5e2651 (0x6fa5 bytes) 000226[0] 1789596818.028280 mapwrite 0x916000 thru 0x9256b1 (0xf6b2 bytes) 000229[0] 1789596818.217161 trunc from 0x938000 to 0x4ae503 (0x489afc bytes) 000231[0] 1789596818.256804 trunc from 0x4ae504 to 0x774418 (0x2c5f15 bytes) 000232[1] 1789596818.284311 write 0x6c2000 thru 0x6d0629 (0xe62a bytes) 000233[1] 1789596818.336977 mapread 0x535000 thru 0x538705 (0x3706 bytes) 000234[1] 1789596818.415003 read 0x3fb000 thru 0x4054f3 (0xa4f4 bytes) 000235[1] 1789596818.459869 write 0x287000 thru 0x28757f (0x580 bytes) 000236[0] 1789596818.472365 read_OD 0x3a3000 thru 0x3a5fff (0x3000 bytes) 000237[0] 1789596818.517890 mapread 0x700000 thru 0x702aac (0x2aad bytes) 000238[0] 1789596818.584900 punch from 0x20d7db to 0x217fad (0xa7d3 bytes) 000239[0] 1789596818.650708 read_OD 0x68f000 thru 0x697fff (0x9000 bytes) 000240[1] 1789596818.681163 punch from 0x2363f5 to 0x23cb6e (0x677a bytes) 000241[0] 1789596818.773959 punch from 0x320e6e to 0x3227ad (0x1940 bytes) 000243[1] 1789596818.900999 read 0x3f7000 thru 0x3fc2d5 (0x52d6 bytes) 000244[0] 1789596818.939640 trunc from 0x774419 to 0x0618b4 (0x712b64 bytes) 000245[1] 1789596819.015902 read 0x03d000 thru 0x03d3d1 (0x3d2 bytes) 000247[0] 1789596819.056076 write_OD 0x547000 thru 0x553fff (0xd000 bytes) 000251[0] 1789596819.116010 read_OD 0x19f000 thru 0x1abfff (0xd000 bytes) 000252[0] 1789596819.126272 mapread 0x52b000 thru 0x52b258 (0x259 bytes) 000254[1] 1789596819.136024 read 0x0c4000 thru 0x0cbd08 (0x7d09 bytes) 000258[0] 1789596819.144294 read_OD 0x274000 thru 0x279fff (0x6000 bytes) 000260[0] 1789596819.153768 punch from 0x4ff2d3 to 0x50751e (0x824c bytes) 000262[0] 1789596819.193417 mapread 0x513000 thru 0x516ad9 (0x3ada bytes) 000265[0] 1789596819.273997 read_OD 0x17e000 thru 0x18afff (0xd000 bytes) 000266[1] 1789596819.289381 read 0x056000 thru 0x058ca4 (0x2ca5 bytes) 000267[0] 1789596819.296615 read_OD 0x089000 thru 0x08bfff (0x3000 bytes) 000268[1] 1789596819.302714 mapread 0x11d000 thru 0x121d8c (0x4d8d bytes) 000271[1] 1789596819.360832 punch from 0x19c52f to 0x1ab35b (0xee2d bytes) 000272[1] 1789596819.387680 mapread 0x1d0000 thru 0x1d79da (0x79db bytes) 000273[0] 1789596819.396382 trunc from 0x554000 to 0x21ed80 (0x33527f bytes) 000275[1] 1789596819.472741 trunc from 0x21ed81 to 0x0eaa46 (0x13433a bytes) 000279[1] 1789596819.524988 write 0x238000 thru 0x240f23 (0x8f24 bytes) 000280[0] 1789596819.526663 mapread 0x1c7000 thru 0x1cc5c3 (0x55c4 bytes) 000281[1] 1789596819.590734 punch from 0x16b870 to 0x179565 (0xdcf6 bytes) 000283[0] 1789596819.622916 write_OD 0x9b3000 thru 0x9bcfff (0xa000 bytes) 000284[0] 1789596819.685188 mapwrite 0x64a000 thru 0x65771f (0xd720 bytes) 000285[1] 1789596819.878811 mapwrite 0x26a000 thru 0x270b33 (0x6b34 bytes) 000286[1] 1789596820.011933 mapread 0x73e000 thru 0x73e109 (0x10a bytes) 000288[1] 1789596820.090753 mapread 0x92b000 thru 0x92b5dc (0x5dd bytes) 000289[0] 1789596820.097299 punch from 0x8435ea to 0x845777 (0x218e bytes) 000291[1] 1789596820.134115 read 0x75e000 thru 0x76b6d3 (0xd6d4 bytes) 000292[0] 1789596820.160144 mapwrite 0x47e000 thru 0x47ebd6 (0xbd7 bytes) 000294[0] 1789596820.283759 read_OD 0x04a000 thru 0x04ffff (0x6000 bytes) 000295[0] 1789596820.309178 read_OD 0x7ec000 thru 0x7ecfff (0x1000 bytes) 000296[0] 1789596820.319469 trunc from 0x9bd000 to 0x266441 (0x756bbe bytes) 000298[1] 1789596820.406320 read 0x14a000 thru 0x1556e0 (0xb6e1 bytes) 000299[1] 1789596820.537784 mapread 0x190000 thru 0x19a18f (0xa190 bytes) 000302[0] 1789596820.584478 mapread 0x22c000 thru 0x22f733 (0x3734 bytes) 000303[0] 1789596820.630421 read_OD 0x087000 thru 0x08efff (0x8000 bytes) 000304[0] 1789596820.653675 mapread 0x14a000 thru 0x14dffd (0x3ffe bytes) 000305[0] 1789596820.677346 punch from 0x11f65b to 0x12df6b (0xe911 bytes) 000306[0] 1789596820.731937 write_OD 0x6b9000 thru 0x6bbfff (0x3000 bytes) 000308[1] 1789596820.794338 mapwrite 0x31f000 thru 0x32a644 (0xb645 bytes) 000311[1] 1789596820.941009 punch from 0x660c8f to 0x6654bc (0x482e bytes) 000312[1] 1789596820.947167 trunc from 0x6bc000 to 0x6904f8 (0x2bb07 bytes) 000316[0] 1789596821.082225 mapread 0x521000 thru 0x52ffd8 (0xefd9 bytes) 000317[0] 1789596821.158357 mapwrite 0x68d000 thru 0x69b3b5 (0xe3b6 bytes) 000322[1] 1789596821.311059 punch from 0x39a233 to 0x3a321e (0x8fec bytes) 000324[1] 1789596821.352712 mapread 0x269000 thru 0x26ebf7 (0x5bf8 bytes) 000326[0] 1789596821.398761 trunc from 0x69b3b6 to 0x0ccc92 (0x5ce723 bytes) 000327[0] 1789596821.438735 mapwrite 0x69f000 thru 0x6a59ba (0x69bb bytes) 000329[1] 1789596821.631291 write 0x790000 thru 0x79ffc9 (0xffca bytes) 000330[1] 1789596821.656802 mapwrite 0x1bb000 thru 0x1be7dc (0x37dd bytes) truncating to largest ever: 0x958072 000331[0] 1789596821.775422 trunc from 0x79ffca to 0x958071 (0x1b80a8 bytes) 000334[0] 1789596821.844400 punch from 0x647adc to 0x656303 (0xe828 bytes) 000336[1] 1789596821.851940 punch from 0x35f845 to 0x36c01d (0xc7d9 bytes) 000337[1] 1789596821.891540 mapread 0x8fc000 thru 0x905e4b (0x9e4c bytes) 000339[0] 1789596821.966469 mapread 0x45c000 thru 0x468216 (0xc217 bytes) 000340[1] 1789596822.041347 mapwrite 0x7bc000 thru 0x7c3dff (0x7e00 bytes) 000341[0] 1789596822.293289 punch from 0x65d0f3 to 0x665d26 (0x8c34 bytes) 000342[0] 1789596822.360764 punch from 0x8255b7 to 0x82aa26 (0x5470 bytes) 000343[1] 1789596822.369165 write 0x2e2000 thru 0x2e7632 (0x5633 bytes) 000345[1] 1789596822.409131 mapwrite 0x89f000 thru 0x8a7f3b (0x8f3c bytes) 000346[0] 1789596822.599360 punch from 0x0c91ec to 0x0ca717 (0x152c bytes) 000347[1] 1789596822.683318 write 0x67f000 thru 0x689341 (0xa342 bytes) 000348[1] 1789596822.734330 read 0x72c000 thru 0x7344ea (0x84eb bytes) 000349[0] 1789596822.746632 mapwrite 0x3ac000 thru 0x3b74dd (0xb4de bytes) 000352[0] 1789596822.907929 mapwrite 0x9d2000 thru 0x9de54c (0xc54d bytes) 000354[0] 1789596823.025075 punch from 0x8933c3 to 0x89a931 (0x756f bytes) 000355[0] 1789596823.041192 mapwrite 0x9ae000 thru 0x9aeefa (0xefb bytes) 000356[1] 1789596823.137976 trunc from 0x9de54d to 0x5be2ca (0x420282 bytes) 000360[0] 1789596823.238838 mapwrite 0x1b1000 thru 0x1bf299 (0xe29a bytes) 000361[0] 1789596823.430552 read_OD 0x59f000 thru 0x5a8fff (0xa000 bytes) 000366[0] 1789596823.469289 trunc from 0x5be2cb to 0x8c7def (0x309b25 bytes) 000367[1] 1789596823.558581 mapread 0x285000 thru 0x289d02 (0x4d03 bytes) 000368[1] 1789596823.654763 punch from 0x2627c6 to 0x271aad (0xf2e8 bytes) 000369[1] 1789596823.710770 mapread 0x499000 thru 0x49bcd8 (0x2cd9 bytes) 000370[1] 1789596823.743276 mapwrite 0x312000 thru 0x31c55d (0xa55e bytes) 000372[1] 1789596823.927956 write 0x0fa000 thru 0x101d7b (0x7d7c bytes) 000374[1] 1789596823.936108 punch from 0x2bb598 to 0x2bc375 (0xdde bytes) 000375[1] 1789596823.960452 mapread 0x21f000 thru 0x22aed5 (0xbed6 bytes) 000379[1] 1789596824.002395 mapwrite 0x0b2000 thru 0x0be1d8 (0xc1d9 bytes) 000381[0] 1789596824.222578 punch from 0x067733 to 0x0687e0 (0x10ae bytes) 000386[0] 1789596824.345672 write_OD 0x865000 thru 0x873fff (0xf000 bytes) 000387[0] 1789596824.405691 write_OD 0x8e0000 thru 0x8e9fff (0xa000 bytes) 000390[1] 1789596824.478729 punch from 0x222c4f to 0x230876 (0xdc28 bytes) 000392[1] 1789596824.496092 mapread 0x41b000 thru 0x4226c2 (0x76c3 bytes) 000393[0] 1789596824.525612 mapread 0x24f000 thru 0x256c5b (0x7c5c bytes) 000394[1] 1789596824.567961 mapwrite 0x49b000 thru 0x4a9e40 (0xee41 bytes) 000396[0] 1789596824.652599 mapread 0x39f000 thru 0x39f0cc (0xcd bytes) 000397[1] 1789596824.690006 mapwrite 0x190000 thru 0x198329 (0x832a bytes) 000398[1] 1789596824.910849 trunc from 0x8ea000 to 0x7982b6 (0x151d49 bytes) 000400[1] 1789596824.958034 trunc from 0x7982b7 to 0x71f560 (0x78d56 bytes) 000401[0] 1789596824.975220 read_OD 0x4be000 thru 0x4c5fff (0x8000 bytes) 000404[0] 1789596825.021132 mapwrite 0x6ca000 thru 0x6cd9e6 (0x39e7 bytes) 000405[0] 1789596825.103925 trunc from 0x71f561 to 0x46fe02 (0x2af75e bytes) 000407[0] 1789596825.150109 write_OD 0x17d000 thru 0x18afff (0xe000 bytes) 000408[0] 1789596825.215427 trunc from 0x46fe03 to 0x3971e9 (0xd8c19 bytes) 000409[1] 1789596825.268927 read 0x279000 thru 0x27dac7 (0x4ac8 bytes) 000410[1] 1789596825.392858 write 0x1c8000 thru 0x1cd119 (0x511a bytes) 000412[0] 1789596825.447831 write_OD 0x036000 thru 0x037fff (0x2000 bytes) 000416[1] 1789596825.561435 write 0x6d9000 thru 0x6e54d9 (0xc4da bytes) 000418[1] 1789596825.579362 read 0x2c6000 thru 0x2d1f09 (0xbf0a bytes) 000419[1] 1789596825.590794 read 0x307000 thru 0x313e18 (0xce19 bytes) 000422[0] 1789596825.639496 write_OD 0x942000 thru 0x945fff (0x4000 bytes) 000423[1] 1789596825.771387 mapread 0x769000 thru 0x7720c1 (0x90c2 bytes) truncating to largest ever: 0x997b5a 000428[0] 1789596825.799371 trunc from 0x946000 to 0x997b59 (0x51b5a bytes) 000430[0] 1789596825.875692 punch from 0x4b4f1e to 0x4be6e2 (0x97c5 bytes) 000431[1] 1789596825.896877 write 0x01c000 thru 0x01cae4 (0xae5 bytes) 000433[0] 1789596825.939998 mapread 0x1a9000 thru 0x1aaa50 (0x1a51 bytes) 000434[1] 1789596825.986791 mapwrite 0x57d000 thru 0x58996b (0xc96c bytes) 000435[1] 1789596826.115998 read 0x214000 thru 0x216b99 (0x2b9a bytes) 000438[0] 1789596826.123932 read_OD 0x5aa000 thru 0x5b1fff (0x8000 bytes) 000440[0] 1789596826.167754 punch from 0x798e65 to 0x79a40b (0x15a7 bytes) 000441[1] 1789596826.202078 mapread 0x4b3000 thru 0x4b8781 (0x5782 bytes) 000443[1] 1789596826.276496 punch from 0x0fe436 to 0x10a8b4 (0xc47f bytes) 000444[1] 1789596826.293997 read 0x82d000 thru 0x83b386 (0xe387 bytes) 000445[1] 1789596826.319935 punch from 0x55a0e6 to 0x561d30 (0x7c4b bytes) 000446[0] 1789596826.324684 read_OD 0x1ec000 thru 0x1f9fff (0xe000 bytes) 000447[1] 1789596826.355268 punch from 0x0115e9 to 0x01b3e1 (0x9df9 bytes) 000449[0] 1789596826.385795 write_OD 0x088000 thru 0x093fff (0xc000 bytes) 000451[1] 1789596826.463909 punch from 0x376cf3 to 0x381521 (0xa82f bytes) 000452[0] 1789596826.497152 write_OD 0x192000 thru 0x19afff (0x9000 bytes) 000456[1] 1789596826.602882 mapread 0x21f000 thru 0x228757 (0x9758 bytes) 000459[0] 1789596826.624885 mapwrite 0x2fd000 thru 0x3026ca (0x56cb bytes) 000460[1] 1789596826.738729 mapwrite 0x26e000 thru 0x2791a7 (0xb1a8 bytes) 000461[0] 1789596826.841788 write_OD 0x772000 thru 0x77ffff (0xe000 bytes) 000462[0] 1789596826.906607 mapwrite 0x8ac000 thru 0x8b735f (0xb360 bytes) 000463[1] 1789596826.991547 mapwrite 0x0c5000 thru 0x0ccc71 (0x7c72 bytes) 000465[0] 1789596827.200647 write_OD 0x1d9000 thru 0x1e0fff (0x8000 bytes) 000467[0] 1789596827.291817 mapread 0x5ce000 thru 0x5d1f1b (0x3f1c bytes) 000469[1] 1789596827.317101 mapread 0x70b000 thru 0x719106 (0xe107 bytes) 000470[1] 1789596827.371389 mapread 0x0c1000 thru 0x0c7cc0 (0x6cc1 bytes) 000475[0] 1789596827.417292 trunc from 0x997b5a to 0x2326b1 (0x7654a8 bytes) 000477[1] 1789596827.491670 write 0x1c2000 thru 0x1cb882 (0x9883 bytes) 000478[0] 1789596827.522471 write_OD 0x117000 thru 0x11dfff (0x7000 bytes) 000480[1] 1789596827.582436 trunc from 0x2326b2 to 0x092e3a (0x19f877 bytes) 000481[1] 1789596827.617797 trunc from 0x092e3b to 0x9689f2 (0x8d5bb8 bytes) 000484[0] 1789596827.697888 write_OD 0x5df000 thru 0x5e6fff (0x8000 bytes) 000485[0] 1789596827.771593 write_OD 0x681000 thru 0x686fff (0x6000 bytes) 000488[1] 1789596827.841858 mapwrite 0x2aa000 thru 0x2b9924 (0xf925 bytes) 000490[1] 1789596827.988447 read 0x1fc000 thru 0x20b83a (0xf83b bytes) 000491[0] 1789596828.003047 trunc from 0x9689f3 to 0x724cf9 (0x243cf9 bytes) 000496[0] 1789596828.101088 mapwrite 0x489000 thru 0x48b88e (0x288f bytes) 000499[0] 1789596828.182756 read_OD 0x359000 thru 0x35cfff (0x4000 bytes) 000500[1] 1789596828.206853 mapwrite 0x237000 thru 0x23aa4b (0x3a4c bytes) All operations completed A-OK! PASS 16a (62s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 18:13:54 (1789596834) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.404559 s, 20.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.425869 s, 19.7 MB/s Chance of close/open is 1 in 50 Seed set to 6838 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[1] 1789596839.072268 mapwrite 0x5e5a47 thru 0x5e6e04 (0x13be bytes) 000004[0] 1789596839.327716 mapwrite 0x36705b thru 0x367ab7 (0xa5d bytes) 000005[1] 1789596839.500874 mapread 0x33d014 thru 0x33d961 (0x94e bytes) truncating to largest ever: 0x4cab05 000007[0] 1789596839.584616 trunc from 0x5e6e05 to 0x4cab04 (0x11c300 bytes) 000008[0] 1789596839.678868 mapread 0x172eff thru 0x174837 (0x1939 bytes) 000010[0] 1789596839.737490 mapread 0x16cccf thru 0x16d635 (0x967 bytes) truncating to largest ever: 0x99a8b8 000011[1] 1789596839.757662 trunc from 0x4cab05 to 0x99a8b7 (0x4cfdb3 bytes) 000012[0] 1789596839.925601 write 0x392682 thru 0x392f87 (0x906 bytes) 000013[1] 1789596839.981673 write 0x770b5d thru 0x771c6c (0x1110 bytes) 000015[1] 1789596840.052056 trunc from 0x99a8b8 to 0x76017c (0x23a73b bytes) 000017[0] 1789596840.127845 write 0x421f1c thru 0x4221b2 (0x297 bytes) 000018[0] 1789596840.164805 mapwrite 0x20b1c2 thru 0x20be4b (0xc8a bytes) 000019[0] 1789596840.253951 read 0x24be4d thru 0x24cae0 (0xc94 bytes) 000020[1] 1789596840.279952 mapwrite 0x1933ea thru 0x195216 (0x1e2d bytes) 000021[0] 1789596840.465074 punch from 0x41ddd7 to 0x41e4a8 (0x6d2 bytes) 000023[1] 1789596840.532203 mapwrite 0x356f52 thru 0x357cdc (0xd8b bytes) 000024[0] 1789596840.622415 mapwrite 0x8b544a thru 0x8b6a39 (0x15f0 bytes) 000026[0] 1789596840.808544 mapread 0x57f620 thru 0x580059 (0xa3a bytes) 000027[0] 1789596840.869074 mapwrite 0x08becc thru 0x08c96c (0xaa1 bytes) 000028[0] 1789596840.983063 punch from 0x1b8794 to 0x1b964f (0xebc bytes) 000030[1] 1789596841.092337 write 0x7cf3a0 thru 0x7d0365 (0xfc6 bytes) 000032[0] 1789596841.193450 write 0x6981b0 thru 0x69a012 (0x1e63 bytes) 000033[0] 1789596841.245178 write 0x0f2fae thru 0x0f3468 (0x4bb bytes) 000034[0] 1789596841.309440 punch from 0x65bec6 to 0x65c013 (0x14e bytes) 000035[0] 1789596841.340103 trunc from 0x8b6a3a to 0x48f44b (0x4275ee bytes) 000036[0] 1789596841.428390 trunc from 0x48f44c to 0x854fe7 (0x3c5b9c bytes) 000037[0] 1789596841.492272 mapread 0x76f364 thru 0x7708f5 (0x1592 bytes) 000038[1] 1789596841.517468 write 0x33438f thru 0x335f2e (0x1ba0 bytes) 000039[0] 1789596841.627468 mapread 0x5822e4 thru 0x584191 (0x1eae bytes) 000040[1] 1789596841.689746 mapread 0x80f0e0 thru 0x810056 (0xf77 bytes) 000041[1] 1789596841.733137 read 0x83ae4e thru 0x83bdad (0xf60 bytes) 000043[1] 1789596841.790323 mapread 0x5d7722 thru 0x5d9617 (0x1ef6 bytes) 000046[1] 1789596841.886052 write 0x275e40 thru 0x276f27 (0x10e8 bytes) 000047[1] 1789596841.906664 mapread 0x1f7789 thru 0x1f7c3c (0x4b4 bytes) 000049[0] 1789596841.912871 trunc from 0x854fe8 to 0x333a1e (0x5215c9 bytes) 000051[1] 1789596841.970872 write 0x6dd907 thru 0x6de872 (0xf6c bytes) 000053[0] 1789596842.018344 write 0x8938eb thru 0x8953d8 (0x1aee bytes) 000054[0] 1789596842.037551 punch from 0x21c689 to 0x21e61d (0x1f95 bytes) truncating to largest ever: 0x9cf0a0 000057[0] 1789596842.101594 trunc from 0x8953d9 to 0x9cf09f (0x139cc7 bytes) 000060[1] 1789596842.189775 mapwrite 0x444a41 thru 0x44612e (0x16ee bytes) 000061[0] 1789596842.357959 mapwrite 0x03e912 thru 0x0403f4 (0x1ae3 bytes) 000065[1] 1789596842.580825 mapread 0x8eb25d thru 0x8ebb40 (0x8e4 bytes) 000066[0] 1789596842.614469 punch from 0x7f52a4 to 0x7f724d (0x1faa bytes) 000067[0] 1789596842.651054 read 0x94369f thru 0x9450d0 (0x1a32 bytes) 000069[0] 1789596842.678402 mapwrite 0x9c4dc8 thru 0x9c5796 (0x9cf bytes) 000070[0] 1789596842.775825 write 0x61d66e thru 0x61f1c6 (0x1b59 bytes) 000071[0] 1789596842.788870 read 0x7ec22d thru 0x7edf18 (0x1cec bytes) 000072[0] 1789596842.823628 trunc from 0x9cf0a0 to 0x930832 (0x9e86d bytes) 000073[1] 1789596842.864058 mapread 0x58e902 thru 0x590831 (0x1f30 bytes) 000074[1] 1789596842.961950 write 0x8af1ad thru 0x8afd10 (0xb64 bytes) 000075[1] 1789596842.989295 trunc from 0x930833 to 0x768462 (0x1c83d0 bytes) 000077[1] 1789596843.024434 mapwrite 0x1748fe thru 0x175825 (0xf28 bytes) 000079[0] 1789596843.135012 write 0x51cc87 thru 0x51d170 (0x4ea bytes) 000080[1] 1789596843.202504 trunc from 0x768463 to 0x8065a6 (0x9e144 bytes) 000081[0] 1789596843.278938 mapwrite 0x399cd2 thru 0x399e30 (0x15f bytes) 000082[1] 1789596843.383941 mapread 0x5d5de1 thru 0x5d6682 (0x8a2 bytes) 000083[1] 1789596843.425471 punch from 0x47566f to 0x476282 (0xc14 bytes) 000084[0] 1789596843.447934 mapread 0x178000 thru 0x1799fb (0x19fc bytes) 000085[0] 1789596843.490399 write 0x0a3e29 thru 0x0a553f (0x1717 bytes) 000086[0] 1789596843.514714 mapread 0x24302e thru 0x244b47 (0x1b1a bytes) 000088[1] 1789596843.568121 trunc from 0x8065a7 to 0x7d9fca (0x2c5dc bytes) 000089[1] 1789596843.644751 mapread 0x6358e7 thru 0x6358ed (0x7 bytes) 000090[1] 1789596843.676868 trunc from 0x7d9fcb to 0x8c2c3d (0xe8c73 bytes) 000091[0] 1789596843.755751 read 0x2d7687 thru 0x2d7df8 (0x772 bytes) 000093[0] 1789596843.802370 read 0x29a691 thru 0x29c354 (0x1cc4 bytes) 000098[1] 1789596843.848047 write 0x3a4f61 thru 0x3a6627 (0x16c7 bytes) 000099[1] 1789596843.912313 punch from 0x1f789a to 0x1f9861 (0x1fc8 bytes) 000104[0] 1789596843.959498 read 0x23bdee thru 0x23d836 (0x1a49 bytes) 000107[0] 1789596844.020610 trunc from 0x8c2c3e to 0x978a3e (0xb5e01 bytes) 000110[0] 1789596844.102831 trunc from 0x978a3f to 0x8c18db (0xb7163 bytes) 000111[0] 1789596844.122332 mapwrite 0x9558de thru 0x957452 (0x1b75 bytes) 000112[1] 1789596844.234058 punch from 0x2e8f39 to 0x2e9991 (0xa59 bytes) 000113[0] 1789596844.264440 read 0x2c20b1 thru 0x2c381a (0x176a bytes) 000115[1] 1789596844.322983 trunc from 0x957453 to 0x7d241b (0x185037 bytes) 000116[0] 1789596844.391085 read 0x65695a thru 0x657bc5 (0x126c bytes) 000117[0] 1789596844.416810 mapwrite 0x5e07a3 thru 0x5e2599 (0x1df7 bytes) 000118[0] 1789596844.533213 trunc from 0x7d241c to 0x415035 (0x3bd3e6 bytes) 000119[1] 1789596844.620283 write 0x059219 thru 0x05ad33 (0x1b1b bytes) 000120[1] 1789596844.673754 mapread 0x286be9 thru 0x28709d (0x4b5 bytes) 000122[1] 1789596844.736946 read 0x353bd5 thru 0x3552d0 (0x16fc bytes) 000123[1] 1789596844.779064 mapwrite 0x8db0fc thru 0x8dc1db (0x10e0 bytes) 000124[1] 1789596844.969940 trunc from 0x8dc1dc to 0x2fda6e (0x5de76d bytes) 000125[1] 1789596844.995103 trunc from 0x2fda6f to 0x583d77 (0x286309 bytes) 000126[0] 1789596845.028481 trunc from 0x583d78 to 0x5e317b (0x5f404 bytes) 000127[1] 1789596845.104830 read 0x0f9d04 thru 0x0fa4e6 (0x7e3 bytes) 000128[1] 1789596845.134423 read 0x018f4a thru 0x01981f (0x8d6 bytes) 000129[0] 1789596845.221811 punch from 0x556249 to 0x556e14 (0xbcc bytes) 000130[1] 1789596845.259222 trunc from 0x5e317c to 0x00cf70 (0x5d620b bytes) 000131[0] 1789596845.346785 punch from 0x00b013 to 0x00cc6a (0x1c58 bytes) 000137[1] 1789596845.511616 mapwrite 0x1960ca thru 0x1970d4 (0x100b bytes) 000139[1] 1789596845.691347 mapread 0x0fda00 thru 0x0fe738 (0xd39 bytes) 000140[0] 1789596845.702231 write 0x8a76e3 thru 0x8a834f (0xc6d bytes) 000141[0] 1789596845.753294 mapread 0x2994a8 thru 0x29b189 (0x1ce2 bytes) 000142[1] 1789596845.822959 trunc from 0x8a8350 to 0x67c7fb (0x22bb54 bytes) 000143[1] 1789596845.865123 trunc from 0x67c7fc to 0x5e789f (0x94f5c bytes) 000144[1] 1789596845.898832 mapwrite 0x998366 thru 0x999b55 (0x17f0 bytes) 000145[1] 1789596846.147877 write 0x6113bf thru 0x612ccc (0x190e bytes) 000147[0] 1789596846.154342 punch from 0x50b6c3 to 0x50ca03 (0x1341 bytes) 000151[1] 1789596846.309496 read 0x191eaa thru 0x1932fe (0x1455 bytes) 000152[1] 1789596846.344004 write 0x922d46 thru 0x922f81 (0x23c bytes) 000153[0] 1789596846.359798 mapread 0x38e468 thru 0x3901f6 (0x1d8f bytes) 000154[1] 1789596846.412115 punch from 0x595a66 to 0x596133 (0x6ce bytes) 000157[1] 1789596846.468352 write 0x4852a1 thru 0x486adb (0x183b bytes) 000160[0] 1789596846.530077 mapread 0x40a947 thru 0x40b8ff (0xfb9 bytes) 000166[1] 1789596846.591708 mapwrite 0x8c69c0 thru 0x8c8837 (0x1e78 bytes) 000168[0] 1789596846.692985 trunc from 0x999b56 to 0x5fc03b (0x39db1a bytes) 000170[1] 1789596846.792151 mapwrite 0x5ea4a9 thru 0x5ea799 (0x2f1 bytes) 000172[1] 1789596846.989461 trunc from 0x5fc03c to 0x675027 (0x78fec bytes) 000173[0] 1789596847.039307 trunc from 0x675028 to 0x605c83 (0x6f3a4 bytes) 000174[1] 1789596847.156262 mapwrite 0x895423 thru 0x895fdc (0xbba bytes) 000175[0] 1789596847.360709 mapwrite 0x5077c1 thru 0x507b52 (0x392 bytes) 000176[0] 1789596847.529877 read 0x598c63 thru 0x598f8c (0x32a bytes) 000177[0] 1789596847.594487 write 0x098d2f thru 0x09a965 (0x1c37 bytes) 000180[0] 1789596847.715619 mapwrite 0x77fe59 thru 0x781e0b (0x1fb3 bytes) 000181[1] 1789596847.830301 trunc from 0x895fdd to 0x0792bb (0x81cd21 bytes) 000182[1] 1789596847.904663 write 0x6ee0ee thru 0x6ef783 (0x1696 bytes) 000187[0] 1789596848.015187 read 0x2b0949 thru 0x2b2866 (0x1f1e bytes) 000188[1] 1789596848.064217 punch from 0x4fa4a9 to 0x4fa51f (0x77 bytes) 000189[0] 1789596848.081808 trunc from 0x6ef784 to 0x88ce51 (0x19d6ce bytes) 000191[0] 1789596848.264019 mapread 0x623fe6 thru 0x62484d (0x868 bytes) 000192[1] 1789596848.345005 mapread 0x84d5e9 thru 0x84e121 (0xb39 bytes) 000193[1] 1789596848.418564 trunc from 0x88ce52 to 0x5792ca (0x313b87 bytes) 000194[1] 1789596848.493313 punch from 0x174617 to 0x1760a5 (0x1a8f bytes) 000195[1] 1789596848.528358 write 0x0e5817 thru 0x0e7344 (0x1b2e bytes) 000197[1] 1789596848.543662 read 0x28497c thru 0x28630b (0x1990 bytes) 000200[1] 1789596848.572904 mapread 0x52cdf1 thru 0x52eb33 (0x1d43 bytes) 000204[0] 1789596848.685068 mapread 0x524f58 thru 0x52683b (0x18e4 bytes) 000205[0] 1789596848.764376 read 0x191713 thru 0x192433 (0xd21 bytes) 000209[0] 1789596848.804058 mapwrite 0x4f045f thru 0x4f1cfa (0x189c bytes) 000212[0] 1789596848.909878 trunc from 0x5792cb to 0x608758 (0x8f48e bytes) 000214[0] 1789596848.953426 mapwrite 0x00a253 thru 0x00b5d1 (0x137f bytes) 000215[1] 1789596849.094036 trunc from 0x608759 to 0x3f9982 (0x20edd6 bytes) 000216[0] 1789596849.208662 read 0x1303b3 thru 0x1314f9 (0x1147 bytes) 000217[1] 1789596849.376288 punch from 0x3e647f to 0x3e6b02 (0x684 bytes) 000218[1] 1789596849.412083 punch from 0x09b2cf to 0x09ce4c (0x1b7e bytes) 000219[1] 1789596849.418285 mapwrite 0x3a9e0c thru 0x3aa66e (0x863 bytes) 000220[1] 1789596849.553849 trunc from 0x3f9983 to 0x8d6d8f (0x4dd40d bytes) 000221[1] 1789596849.609148 read 0x001e22 thru 0x002b66 (0xd45 bytes) 000222[0] 1789596849.647039 write 0x2f0413 thru 0x2f0d0a (0x8f8 bytes) 000223[1] 1789596849.695238 mapwrite 0x859b3b thru 0x85a8b7 (0xd7d bytes) 000224[0] 1789596849.805064 mapread 0x63492d thru 0x634afc (0x1d0 bytes) 000225[1] 1789596850.017482 write 0x670531 thru 0x6722f9 (0x1dc9 bytes) 000227[0] 1789596850.077180 mapwrite 0x391704 thru 0x3927c1 (0x10be bytes) 000228[1] 1789596850.323272 write 0x5292b9 thru 0x52aa01 (0x1749 bytes) 000230[1] 1789596850.429975 mapwrite 0x4e0b57 thru 0x4e1c38 (0x10e2 bytes) 000231[0] 1789596850.563881 read 0x44f336 thru 0x44fadd (0x7a8 bytes) 000232[0] 1789596850.605494 mapread 0x120ac7 thru 0x120ff3 (0x52d bytes) 000234[0] 1789596850.627419 mapwrite 0x7ac38d thru 0x7ac3f2 (0x66 bytes) 000236[0] 1789596850.740711 write 0x70fe15 thru 0x71089e (0xa8a bytes) 000237[1] 1789596850.780397 read 0x1f2126 thru 0x1f25c5 (0x4a0 bytes) 000238[1] 1789596850.803345 mapread 0x26df9f thru 0x26e354 (0x3b6 bytes) 000239[1] 1789596850.817910 mapwrite 0x324021 thru 0x324c13 (0xbf3 bytes) 000240[1] 1789596850.907551 mapwrite 0x21e538 thru 0x21e6b8 (0x181 bytes) 000241[0] 1789596850.987422 mapread 0x1e3550 thru 0x1e4381 (0xe32 bytes) 000242[0] 1789596851.016403 mapread 0x641ae9 thru 0x6421d6 (0x6ee bytes) 000245[0] 1789596851.046797 read 0x8b92ef thru 0x8ba80f (0x1521 bytes) 000247[1] 1789596851.057930 mapread 0x1f6d4c thru 0x1f848d (0x1742 bytes) 000250[0] 1789596851.123672 trunc from 0x8d6d90 to 0x342277 (0x594b18 bytes) 000251[1] 1789596851.162648 trunc from 0x342278 to 0x862bae (0x520937 bytes) 000253[0] 1789596851.252301 write 0x6b3c15 thru 0x6b4870 (0xc5c bytes) 000254[1] 1789596851.290297 punch from 0x7631da to 0x76446e (0x1295 bytes) 000255[1] 1789596851.331822 mapread 0x2d31f3 thru 0x2d3375 (0x183 bytes) 000256[1] 1789596851.356285 mapwrite 0x4d4fb3 thru 0x4d5a4f (0xa9d bytes) 000257[1] 1789596851.425333 mapread 0x6ae88c thru 0x6af207 (0x97c bytes) 000258[0] 1789596851.455737 punch from 0x356328 to 0x3579c5 (0x169e bytes) 000261[1] 1789596851.511226 mapwrite 0x937a46 thru 0x937d3c (0x2f7 bytes) 000263[0] 1789596851.613503 mapwrite 0x9ced20 thru 0x9cf185 (0x466 bytes) 000265[0] 1789596851.786814 punch from 0x775d63 to 0x7764f2 (0x790 bytes) 000266[0] 1789596851.818435 trunc from 0x9cf186 to 0x79ef06 (0x23027f bytes) 000268[0] 1789596851.889743 trunc from 0x79ef07 to 0x0bf6d8 (0x6df82e bytes) 000270[1] 1789596851.964567 trunc from 0x0bf6d9 to 0x84e553 (0x78ee7b bytes) 000271[0] 1789596852.032460 mapread 0x065d87 thru 0x065d8c (0x6 bytes) 000274[0] 1789596852.100666 mapwrite 0x112266 thru 0x112709 (0x4a4 bytes) 000275[1] 1789596852.202123 mapwrite 0x201ebc thru 0x203d3c (0x1e81 bytes) 000276[1] 1789596852.310978 mapread 0x309686 thru 0x30a476 (0xdf1 bytes) 000278[1] 1789596852.373411 punch from 0x25fe0b to 0x2615f3 (0x17e9 bytes) 000279[0] 1789596852.404951 read 0x6b7e2d thru 0x6b8507 (0x6db bytes) 000280[1] 1789596852.446426 write 0x3c2a30 thru 0x3c3450 (0xa21 bytes) 000281[0] 1789596852.490245 mapwrite 0x356b11 thru 0x3580a7 (0x1597 bytes) 000283[1] 1789596852.640317 punch from 0x3cb842 to 0x3cc57d (0xd3c bytes) 000287[0] 1789596852.672657 mapread 0x162c8e thru 0x1638bc (0xc2f bytes) 000289[1] 1789596852.713090 mapwrite 0x4a0e07 thru 0x4a2377 (0x1571 bytes) 000290[0] 1789596852.811433 punch from 0x712333 to 0x713b52 (0x1820 bytes) 000291[0] 1789596852.839505 mapwrite 0x5cb06a thru 0x5cc535 (0x14cc bytes) 000293[0] 1789596852.926139 read 0x320adb thru 0x32276e (0x1c94 bytes) 000295[1] 1789596852.947248 mapwrite 0x8d6cf6 thru 0x8d7a30 (0xd3b bytes) 000296[1] 1789596853.055544 punch from 0x79e8c8 to 0x7a04a1 (0x1bda bytes) 000298[1] 1789596853.063480 read 0x84608c thru 0x847ef4 (0x1e69 bytes) 000299[0] 1789596853.102391 punch from 0x140bad to 0x1419df (0xe33 bytes) 000300[1] 1789596853.132760 read 0x323502 thru 0x32439e (0xe9d bytes) 000301[0] 1789596853.170670 punch from 0x3bb7c7 to 0x3bcadf (0x1319 bytes) 000303[1] 1789596853.216208 read 0x32d80c thru 0x32e1c4 (0x9b9 bytes) 000305[0] 1789596853.235255 mapread 0x89e6eb thru 0x89fd2a (0x1640 bytes) 000306[1] 1789596853.257175 mapread 0x305b36 thru 0x30748e (0x1959 bytes) 000307[1] 1789596853.285774 trunc from 0x8d7a31 to 0x060daf (0x876c81 bytes) 000308[1] 1789596853.330955 mapread 0x0093e7 thru 0x00941d (0x37 bytes) 000309[0] 1789596853.366149 mapread 0x04f481 thru 0x0504e0 (0x1060 bytes) 000313[1] 1789596853.446796 trunc from 0x060db0 to 0x9457c7 (0x8e4a18 bytes) 000314[1] 1789596853.485335 mapwrite 0x8becb6 thru 0x8bf1b2 (0x4fd bytes) 000316[0] 1789596853.536422 punch from 0x511220 to 0x513098 (0x1e79 bytes) 000317[1] 1789596853.565038 mapwrite 0x0d6773 thru 0x0d6ccc (0x55a bytes) 000319[0] 1789596853.664428 read 0x7f8c78 thru 0x7f92f8 (0x681 bytes) 000320[0] 1789596853.701714 mapwrite 0x0e5ff5 thru 0x0e6d5e (0xd6a bytes) 000322[0] 1789596853.805271 punch from 0x89fc29 to 0x8a0faf (0x1387 bytes) 000325[1] 1789596853.858040 read 0x928af1 thru 0x928f99 (0x4a9 bytes) 000327[0] 1789596853.886336 trunc from 0x9457c8 to 0x18af90 (0x7ba837 bytes) 000328[1] 1789596853.940497 mapwrite 0x00e8aa thru 0x00f23d (0x994 bytes) 000329[0] 1789596854.069166 mapread 0x06388f thru 0x064391 (0xb03 bytes) 000332[1] 1789596854.119183 punch from 0x0a3f98 to 0x0a57ca (0x1833 bytes) 000335[0] 1789596854.196966 write 0x7749ba thru 0x775858 (0xe9f bytes) 000336[1] 1789596854.234371 punch from 0x030dcb to 0x032a79 (0x1caf bytes) 000337[0] 1789596854.261548 read 0x1f4aab thru 0x1f672b (0x1c81 bytes) 000338[0] 1789596854.306385 mapwrite 0x1b28e0 thru 0x1b47da (0x1efb bytes) 000340[0] 1789596854.424822 write 0x24f727 thru 0x24fcde (0x5b8 bytes) 000343[0] 1789596854.522555 mapwrite 0x7d1bd1 thru 0x7d1eb9 (0x2e9 bytes) 000344[0] 1789596854.680429 read 0x4f283e thru 0x4f38e9 (0x10ac bytes) 000351[0] 1789596854.762130 mapwrite 0x498cf3 thru 0x49a14b (0x1459 bytes) 000352[1] 1789596854.887058 read 0x3d73f9 thru 0x3d897a (0x1582 bytes) 000353[0] 1789596854.915019 write 0x7e5b0d thru 0x7e6d38 (0x122c bytes) 000354[0] 1789596854.928627 trunc from 0x7e6d39 to 0x338bcf (0x4ae169 bytes) 000356[1] 1789596855.013837 trunc from 0x338bd0 to 0x937694 (0x5feac5 bytes) 000357[0] 1789596855.081403 trunc from 0x937695 to 0x2fc5a3 (0x63b0f1 bytes) 000358[0] 1789596855.115926 write 0x8fcb78 thru 0x8fd3d0 (0x859 bytes) 000363[1] 1789596855.170376 mapwrite 0x92a2c4 thru 0x92ae6d (0xbaa bytes) 000364[1] 1789596855.292104 punch from 0x2a3919 to 0x2a5404 (0x1aec bytes) 000368[1] 1789596855.320871 mapwrite 0x29c717 thru 0x29d9b6 (0x12a0 bytes) 000369[1] 1789596855.396782 mapread 0x6928bf thru 0x6944e8 (0x1c2a bytes) 000370[1] 1789596855.475380 read 0x3ab0f8 thru 0x3abbde (0xae7 bytes) 000371[1] 1789596855.481189 punch from 0x0d6fda to 0x0d7294 (0x2bb bytes) 000372[1] 1789596855.491868 punch from 0x1a74e9 to 0x1a8c6e (0x1786 bytes) 000376[1] 1789596855.514111 mapwrite 0x86d033 thru 0x86d97f (0x94d bytes) 000379[1] 1789596855.611396 punch from 0x8bf3ab to 0x8bfa8e (0x6e4 bytes) 000381[1] 1789596855.643268 mapwrite 0x53ac01 thru 0x53c416 (0x1816 bytes) 000382[1] 1789596855.749572 write 0x749d9c thru 0x74b80f (0x1a74 bytes) 000384[0] 1789596855.785309 mapread 0x6b7ad2 thru 0x6b8202 (0x731 bytes) 000385[0] 1789596855.883662 read 0x400973 thru 0x400c2c (0x2ba bytes) 000386[0] 1789596855.939695 punch from 0x315c73 to 0x3169f9 (0xd87 bytes) 000387[0] 1789596855.963463 write 0x878d6c thru 0x87abae (0x1e43 bytes) 000389[1] 1789596855.998892 write 0x3b9264 thru 0x3bb148 (0x1ee5 bytes) 000390[0] 1789596856.043104 punch from 0x7bafdf to 0x7bca90 (0x1ab2 bytes) 000391[0] 1789596856.079424 trunc from 0x92ae6e to 0x85ce53 (0xce01a bytes) 000392[1] 1789596856.190350 read 0x5878d5 thru 0x588c70 (0x139c bytes) 000394[0] 1789596856.263374 mapwrite 0x32ce9e thru 0x32e4b0 (0x1613 bytes) 000397[1] 1789596856.399390 trunc from 0x85ce54 to 0x5cfc4f (0x28d204 bytes) 000399[1] 1789596856.494091 mapread 0x225bfb thru 0x2263b9 (0x7bf bytes) 000400[1] 1789596856.525895 read 0x5a33ed thru 0x5a3f5c (0xb70 bytes) 000401[1] 1789596856.551375 write 0x734b14 thru 0x7364a6 (0x1993 bytes) 000402[0] 1789596856.553114 read 0x1c20ac thru 0x1c2a92 (0x9e7 bytes) 000403[1] 1789596856.600691 read 0x4b726e thru 0x4b75df (0x372 bytes) 000412[0] 1789596856.715847 mapwrite 0x32529d thru 0x325f57 (0xcbb bytes) 000414[0] 1789596856.899557 mapwrite 0x55b099 thru 0x55bbf7 (0xb5f bytes) 000416[1] 1789596857.038205 mapread 0x04ad5a thru 0x04ae63 (0x10a bytes) 000419[1] 1789596857.106379 trunc from 0x7364a7 to 0x21a4e3 (0x51bfc3 bytes) 000420[1] 1789596857.179392 mapwrite 0x4c22ba thru 0x4c3dbb (0x1b02 bytes) 000422[1] 1789596857.260192 trunc from 0x4c3dbc to 0x4641cb (0x5fbf0 bytes) 000423[1] 1789596857.300482 trunc from 0x4641cc to 0x6361e9 (0x1d201e bytes) 000425[1] 1789596857.367884 write 0x8987ca thru 0x899e6d (0x16a4 bytes) 000426[1] 1789596857.369808 punch from 0x12a8f1 to 0x12ab4b (0x25b bytes) 000427[0] 1789596857.404911 punch from 0x44c073 to 0x44d84a (0x17d8 bytes) 000430[0] 1789596857.517974 mapwrite 0x22b454 thru 0x22c064 (0xc11 bytes) 000431[0] 1789596857.627616 read 0x63dcc9 thru 0x63ea40 (0xd78 bytes) 000434[1] 1789596857.688250 mapwrite 0x77b090 thru 0x77b46c (0x3dd bytes) 000435[1] 1789596857.819309 trunc from 0x899e6e to 0x84193a (0x58533 bytes) 000438[0] 1789596857.885847 mapwrite 0x6aa162 thru 0x6ab5d7 (0x1476 bytes) 000439[1] 1789596858.044153 trunc from 0x84193b to 0x9c12b6 (0x17f97c bytes) 000440[1] 1789596858.134870 mapwrite 0x36b664 thru 0x36c6b4 (0x1051 bytes) 000442[1] 1789596858.311234 write 0x8a275e thru 0x8a3bc5 (0x1468 bytes) 000445[0] 1789596858.380557 mapread 0x7ed8fe thru 0x7ef272 (0x1975 bytes) 000449[0] 1789596858.534973 write 0x9dbe30 thru 0x9dc291 (0x462 bytes) 000450[1] 1789596858.592447 write 0x49675b thru 0x498133 (0x19d9 bytes) 000452[1] 1789596858.660090 punch from 0x9bed43 to 0x9bf94d (0xc0b bytes) 000453[0] 1789596858.716069 mapread 0x98a961 thru 0x98b593 (0xc33 bytes) 000454[1] 1789596858.768883 punch from 0x74c76a to 0x74e2c3 (0x1b5a bytes) 000456[1] 1789596858.816036 mapwrite 0x202b5b thru 0x203d23 (0x11c9 bytes) 000457[1] 1789596858.935975 trunc from 0x9dc292 to 0x0b1075 (0x92b21c bytes) 000459[1] 1789596858.990955 read 0x08b5aa thru 0x08c093 (0xaea bytes) 000460[1] 1789596859.026632 write 0x0199a1 thru 0x01a21d (0x87d bytes) 000461[0] 1789596859.037408 write 0x953241 thru 0x955002 (0x1dc2 bytes) 000462[1] 1789596859.071380 read 0x82dbef thru 0x82ec5f (0x1071 bytes) 000463[0] 1789596859.260761 punch from 0x841160 to 0x8418c9 (0x76a bytes) 000464[0] 1789596859.298313 write 0x770d15 thru 0x7719dc (0xcc8 bytes) 000465[0] 1789596859.376716 read 0x096b0e thru 0x098af1 (0x1fe4 bytes) 000466[1] 1789596859.390578 mapread 0x266486 thru 0x266d31 (0x8ac bytes) 000468[1] 1789596859.453026 read 0x369acb thru 0x36abfd (0x1133 bytes) 000469[1] 1789596859.465668 mapread 0x31de7d thru 0x31e33f (0x4c3 bytes) 000470[1] 1789596859.494030 trunc from 0x955003 to 0x74e0ec (0x206f16 bytes) 000471[0] 1789596859.552402 mapwrite 0x4dac82 thru 0x4dbd1a (0x1099 bytes) 000473[0] 1789596859.656830 write 0x99f1f9 thru 0x99f995 (0x79d bytes) 000474[1] 1789596859.671330 mapwrite 0x399b15 thru 0x39a7ea (0xcd6 bytes) 000475[1] 1789596859.778572 write 0x80a75e thru 0x80af21 (0x7c4 bytes) 000476[1] 1789596859.797994 read 0x4b4209 thru 0x4b4bf3 (0x9eb bytes) 000477[0] 1789596859.827761 trunc from 0x99f996 to 0x0e8229 (0x8b776c bytes) 000483[0] 1789596859.976972 punch from 0x0c2d96 to 0x0c4512 (0x177d bytes) 000484[0] 1789596859.983412 mapread 0x016e05 thru 0x01758e (0x78a bytes) 000486[1] 1789596860.028205 mapwrite 0x0e1b09 thru 0x0e20cc (0x5c4 bytes) 000487[1] 1789596860.150066 mapread 0x016608 thru 0x0180fa (0x1af3 bytes) 000489[0] 1789596860.225584 punch from 0x0a8c36 to 0x0a94b7 (0x882 bytes) 000490[0] 1789596860.242648 write 0x08f87f thru 0x08ff9d (0x71f bytes) All operations completed A-OK! PASS 16b (33s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 18:14:26 (1789596866) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 18:14:31 (1789596871) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.481437 s, 17.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.570276 s, 14.7 MB/s ldlm.namespaces.MGC192.168.206.133@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff936444f00800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff936448070000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff936448070000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff936448070000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.8245 s, 7.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.6964 s, 28.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.8823 s, 36.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.3283 s, 16.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.57846 s, 12.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.84913 s, 27.2 MB/s PASS 16d (55s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 18:15:26 (1789596926) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.371752 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185171 s, 5.7 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 18:15:33 (1789596933) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 18:16:01 (1789596961) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 18:16:30 (1789596990) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 18:16:38 (1789596998) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 18:16:46 (1789597006) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.63547 s, 16.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.5647 s, 16.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.3362 s, 7.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.86248 s, 14.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.68334 s, 9.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.47361 s, 10.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.36041 s, 7.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.12475 s, 12.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.25876 s, 11.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.51223 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.71697 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.24619 s, 11.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.4723 s, 10.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.17472 s, 8.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.28773 s, 8.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.04089 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.5356 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.04464 s, 6.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.2058 s, 6.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (44s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 18:17:30 (1789597050) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Wed Sep 16 18:18:20 EDT 2026 + fsx 2 done: rc=0 Wed Sep 16 18:18:20 EDT 2026 + fsx 3 done: rc=0 Wed Sep 16 18:18:20 EDT 2026 + fsx 4 done: rc=0 Wed Sep 16 18:18:20 EDT 2026 + fsx 5 done: rc=0 Wed Sep 16 18:18:20 EDT 2026 PASS 16k (60s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 18:18:30 (1789597110) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 18:18:42 (1789597122) mmap test1: basic mmap operation (PASS, 0.1466s) mmap test2: MAP_PRIVATE not write back (PASS, 0.070207s) mmap test3: concurrent mmap ops on two nodes (PASS, 11.659s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5263s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.14406s) mmap test6: check mmap write/read content on two nodes (PASS, 0.32499s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.4899s) mmap test11: multi-thread shared mmap (PASS, 1.5553s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1033s) PASS 18 (31s) == sanityn test 19: test concurrent uncached read races ========================================================================= 18:19:12 (1789597152) oleg633-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg633-client: oleg633-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 18:19:17 (1789597157) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 18:19:24 (1789597164) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 18:19:33 (1789597173) now is 1789597175 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 new atime is 1789597236 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 18:20:43 (1789597243) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 18:20:50 (1789597250) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 18:20:56 (1789597256) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 18:21:03 (1789597263) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 18:21:07 (1789597267) utime: good mknod times 1789597268 <= 1789597269 <= 1789597269 for /mnt/lustre/f26a utime: good mknod times 1789597268 <= 1789597269 <= 1789597269 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 18:21:15 (1789597275) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 18:21:23 (1789597283) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-16 18:21:25.000000000 -0400 Birth: 2026-09-16 18:21:24.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-16 18:21:25.000000000 -0400 Birth: 2026-09-16 18:21:24.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 18:21:32 (1789597292) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 18:21:41 (1789597301) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 18:21:49 (1789597309) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg633-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.133@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 18:22:10 (1789597330) -rw-r--r-- 1 root root 0 Sep 16 18:22 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 18:22:19 (1789597339) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43191 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 18:22:26 (1789597346) PASS 31t (6s) == sanityn test 32b: lockless i/o ======================== 18:22:32 (1789597352) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 18:22:35 (1789597355) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49085 s, 35.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.38769 s, 37.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.84576 s, 28.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.135 s, 24.6 MB/s contention_seconds 4, contention_events 15 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.696 s, 30.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.91915 s, 27.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.14095 s, 46.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.22625 s, 42.8 MB/s PASS 32c (29s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 18:23:06 (1789597386) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 18:23:10 (1789597390) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 18:23:13 (1789597393) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 18:23:16 (1789597396) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 18:23:19 (1789597399) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff936444f00800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936444f00800.ost_server_uuid in IDLE state after 0 sec oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff936444f00800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff936444f00800.ost_server_uuid in IDLE state after 0 sec oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff936444f00800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936444f00800.ost_server_uuid in IDLE state after 0 sec oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff936444f00800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff936444f00800.ost_server_uuid in IDLE state after 0 sec oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff936444f00800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936444f00800.ost_server_uuid in IDLE state after 0 sec oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff936444f00800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff936444f00800.ost_server_uuid in IDLE state after 0 sec PASS 34 (135s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 18:25:34 (1789597534) fail_loc=0x80000317 Race attempt 0 Wait for 50495 50522 for 60 sec... - open/close 662 (time 1789597551.04 total 10.01 last 66.12) - open/close 1347 (time 1789597561.05 total 20.01 last 68.49) - open/close 2095 (time 1789597571.05 total 30.01 last 74.79) - open/close 2846 (time 1789597581.06 total 40.02 last 75.04) - open/close 3598 (time 1789597591.07 total 50.04 last 75.10) total: 4000 open/close in 55.83 seconds: 71.65 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.133@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.206.133@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 18:26:46 (1789597606) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg633-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.45036 s, 36.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4484 s, 36.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.39154 s, 37.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7493632) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41014 s, 37.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.69784 s, 30.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.36666 s, 38.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48971 s, 35.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7493632) after_dd(7444480) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.13411 s, 46.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.41258 s, 37.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7440384) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.85086 s, 28.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29123 s, 40.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7446528) after(7497728) PASS 36 (321s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 18:32:07 (1789597927) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 - create 1346 (time 1789597941.65 total 10.00 last 134.57) - create 2629 (time 1789597951.65 total 20.01 last 128.25) - create 3957 (time 1789597961.66 total 30.01 last 132.73) - create 5210 (time 1789597971.67 total 40.02 last 125.20) - create 6491 (time 1789597981.67 total 50.02 last 128.03) - create 7938 (time 1789597991.68 total 60.03 last 144.60) - create 9334 (time 1789598001.68 total 70.04 last 139.55) total: 10000 create in 75.07 seconds: 133.21 ops/second PASS 37 (87s) == sanityn test 39a: file mtime does not change after rename ========================================================== 18:33:34 (1789598014) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 18:33:41 (1789598021) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 18:33:49 (1789598029) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 18:33:57 (1789598037) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 18:34:05 (1789598045) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 18:34:23 (1789598063) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (19s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 18:34:43 (1789598083) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (19s) == sanityn test 40e: pdirops: rename and others ======================================================================== 18:35:02 (1789598102) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 18:35:19 (1789598119) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 18:35:33 (1789598133) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 18:35:47 (1789598147) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 18:36:01 (1789598161) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 18:36:14 (1789598174) Conflict PASS 41e (12s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 18:36:26 (1789598186) Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 18:36:39 (1789598199) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 18:36:53 (1789598213) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 18:37:07 (1789598227) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1064s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 18:54:52 (1789599292) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 18:55:05 (1789599305) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 18:55:18 (1789599318) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 18:55:32 (1789599332) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 18:55:45 (1789599345) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 18:55:59 (1789599359) Conflict PASS 42f (12s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 18:56:11 (1789599371) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 18:56:24 (1789599384) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 18:56:37 (1789599397) PASS 43a (76s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 18:57:53 (1789599473) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 18:58:07 (1789599487) Conflict PASS 43c (13s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 18:58:20 (1789599500) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 18:58:34 (1789599514) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 18:58:47 (1789599527) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 18:59:00 (1789599540) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 18:59:14 (1789599554) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 18:59:29 (1789599569) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 18:59:33 (1789599573) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (138s) == sanityn test 43k: unlink vs create ==================== 19:01:51 (1789599711) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1142s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 19:20:53 (1789600853) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Sep 16 19:21:01 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 19:21:07 (1789600867) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 19:21:21 (1789600881) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 19:21:36 (1789600896) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 19:21:49 (1789600909) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 19:22:04 (1789600924) Conflict PASS 44f (13s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 19:22:17 (1789600937) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 19:22:29 (1789600949) Conflict PASS 44h (12s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 19:22:42 (1789600962) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 19:22:45 (1789600965) PASS 45a (123s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 19:24:49 (1789601089) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 19:25:02 (1789601102) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 19:25:17 (1789601117) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (15s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 19:25:32 (1789601132) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 19:25:48 (1789601148) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (15s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 19:26:02 (1789601162) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 19:26:16 (1789601176) Conflict PASS 45h (12s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 19:26:28 (1789601188) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 19:26:31 (1789601191) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1146s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 19:45:37 (1789602337) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 19:45:50 (1789602350) open: File exists Conflict PASS 46b (14s) == sanityn test 46c: pdirops: link vs link ======================================================================== 19:46:04 (1789602364) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 19:46:17 (1789602377) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 19:46:30 (1789602390) Conflict PASS 46e (12s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 19:46:42 (1789602402) Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 19:46:56 (1789602416) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 19:47:08 (1789602428) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 19:47:22 (1789602442) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 19:47:26 (1789602446) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 19:47:29 (1789602449) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 19:47:33 (1789602453) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 19:47:36 (1789602456) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 19:47:39 (1789602459) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 19:47:43 (1789602463) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 19:47:47 (1789602467) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 19:47:50 (1789602470) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.120167 s, 85.2 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 19:48:03 (1789602483) 0+1 records in 0+1 records out 158 bytes copied, 0.0380816 s, 4.1 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218828) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 19:48:12 (1789602492) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0161854 s, 63.3 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857797 s, 12.2 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 19:48:34 (1789602514) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.03682 s, 27.8 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 19:48:46 (1789602526) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573944 s, 18.3 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (7s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 19:48:54 (1789602534) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2448 0x990 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2448 0x990 0x280000400 read lease(1) has applied. PASS 51e (8s) == sanityn test 54: rename locking ======================= 19:49:02 (1789602542) llite.lustre-ffff936444f00800.xattr_cache=1 llite.lustre-ffff936448070000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 19:49:05.000000000 -0400 Modify: 2026-09-16 19:49:05.000000000 -0400 Change: 2026-09-16 19:49:05.000000000 -0400 Birth: 2026-09-16 19:49:04.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 19:49:11.000000000 -0400 Modify: 2026-09-16 19:49:12.000000000 -0400 Change: 2026-09-16 19:49:12.000000000 -0400 Birth: 2026-09-16 19:49:10.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 19:49:18.000000000 -0400 Modify: 2026-09-16 19:49:18.000000000 -0400 Change: 2026-09-16 19:49:18.000000000 -0400 Birth: 2026-09-16 19:49:17.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 19:49:18.000000000 -0400 Modify: 2026-09-16 19:49:18.000000000 -0400 Change: 2026-09-16 19:49:18.000000000 -0400 Birth: 2026-09-16 19:49:17.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 19:49:24.000000000 -0400 Modify: 2026-09-16 19:49:25.000000000 -0400 Change: 2026-09-16 19:49:25.000000000 -0400 Birth: 2026-09-16 19:49:23.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 19:49:24.000000000 -0400 Modify: 2026-09-16 19:49:25.000000000 -0400 Change: 2026-09-16 19:49:25.000000000 -0400 Birth: 2026-09-16 19:49:23.000000000 -0400 PASS 54 (33s) == sanityn test 55a: rename vs unlink target dir ========= 19:49:35 (1789602575) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 19:49:49 (1789602589) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 19:50:03 (1789602603) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 19:50:23 (1789602623) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 19:50:37 (1789602637) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 19:50:40 (1789602640) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 19:50:59 (1789602659) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (23s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 19:51:22 (1789602682) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789602684 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.554901 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (119s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 19:53:21 (1789602801) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789602808 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.09977 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (126s) == sanityn test 60: Verify data_version behaviour ======== 19:55:27 (1789602927) PASS 60 (9s) == sanityn test 70a: cd directory && rm directory ======== 19:55:37 (1789602937) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 19:55:45 (1789602945) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 19:55:54 (1789602954) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0085877 s, 7.6 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 19:55:58 (1789602958) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0150224 s, 4.4 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 19:56:01 (1789602961) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 19:56:05 (1789602965) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0235288 s, 2.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 19:56:08 (1789602968) llite.lustre-ffff936444f00800.xattr_cache=1 llite.lustre-ffff936448070000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 19:56:15 (1789602975) llite.lustre-ffff936444f00800.xattr_cache=1 llite.lustre-ffff936448070000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (6s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 19:56:21 (1789602981) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0114086 s, 89.8 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0117988 s, 86.8 kB/s 231649: get lock2 231650: get lock1 231649: done 231649: get lock1 231650: done 231650: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231650: cannot get lock2: Resource deadlock avoided 231650: exit rc=0 231649: done 231649: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 19:56:32 (1789602992) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.077043 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395479 s, 26.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232233 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (11s) == sanityn test 76: Verify MDT open_files listing ======== 19:56:43 (1789603003) opening files: ........ closing files: ........ PASS 76 (69s) == sanityn test 77a: check FIFO NRS policy =============== 19:57:52 (1789603072) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.45175 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794606 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905929 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872515 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671539 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339577 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710462 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154806 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123498 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189773 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219916 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231901 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180828 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132939 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164447 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220772 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188196 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384884 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566444 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461368 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594177 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454446 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912601 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480395 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476338 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772099 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783603 s, 13.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673041 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00545202 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114115 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00950545 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00906081 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140654 s, 74.6 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 19:58:01 (1789603081) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.464783 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909824 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631889 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142312 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168586 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171103 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192828 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244482 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197145 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147897 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209729 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109217 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192059 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103738 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292524 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171912 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157766 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118165 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116783 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102605 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858767 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822421 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952466 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097224 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126278 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900164 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144086 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961708 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113159 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973511 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088779 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115819 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165708 s, 6.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.409947 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100187 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1411 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141469 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177538 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128103 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118029 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480217 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181785 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150971 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166403 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180667 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132322 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164791 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160377 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132598 s, 7.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173317 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405302 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019445 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309895 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262321 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220313 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269363 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030177 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381981 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576289 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057772 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656126 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512233 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811021 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509022 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565607 s, 18.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511053 s, 20.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (12s) == sanityn test 77c: check ORR NRS policy ================ 19:58:13 (1789603093) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.320088 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502396 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120137 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140251 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141107 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158255 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195565 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120108 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15784 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708627 s, 14.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273352 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293332 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387141 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356648 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244336 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366606 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40019 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526936 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723727 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244571 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900086 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305926 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962064 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422276 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410055 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644191 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102581 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525137 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524442 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680383 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014523 s, 72.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663832 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105018 s, 10.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.443016 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054668 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523065 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196199 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168541 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221167 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196646 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205485 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194724 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202744 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237607 s, 4.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230801 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234569 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214676 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261208 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23923 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26002 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471458 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353982 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065195 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358402 s, 29.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210325 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252322 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461133 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761244 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488377 s, 21.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576372 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417265 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426902 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794485 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813422 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535678 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585245 s, 17.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.404453 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807217 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101197 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155397 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138947 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12216 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178795 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173387 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191658 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236018 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242867 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22971 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198363 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25495 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357967 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39387 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353837 s, 3.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053381 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157704 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119779 s, 87.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675717 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176386 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111068 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116848 s, 89.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187213 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120318 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158619 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860922 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872679 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113524 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361645 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338007 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231257 s, 45.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (15s) == sanityn test 77d: check TRR nrs policy ================ 19:58:28 (1789603108) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.451608 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274566 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071877 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756516 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130184 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197921 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220448 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219431 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240209 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26039 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294456 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292179 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300386 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276683 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27862 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287741 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357094 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276153 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274598 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294664 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777681 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343262 s, 30.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424736 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446912 s, 23.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497026 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456015 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538909 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586739 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581244 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537332 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543481 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570201 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701122 s, 15.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.62483 s, 26.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639944 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908731 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144181 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11091 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548885 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159296 s, 6.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831856 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957322 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982307 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126355 s, 8.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352574 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21188 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263619 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32182 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266662 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350583 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283597 s, 37.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412725 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760108 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338324 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169851 s, 61.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454696 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015202 s, 69.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507574 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537319 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547511 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565658 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218847 s, 47.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341605 s, 30.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229271 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241905 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014877 s, 70.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.502883 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448901 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506518 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823883 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146933 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17789 s, 5.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186443 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110079 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135747 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262625 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101246 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126067 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17579 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102264 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358231 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173846 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396583 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607749 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216342 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247208 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322427 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740261 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332576 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384649 s, 27.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861662 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864693 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900293 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533739 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056199 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612209 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703098 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709222 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109611 s, 9.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (16s) == sanityn test 77e: check TBF NID nrs policy ============ 19:58:44 (1789603124) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.33@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.404577 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357913 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117003 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126335 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140246 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102781 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147554 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125638 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102392 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148924 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17082 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209889 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208799 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137405 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220233 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179894 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123064 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102886 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119941 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876897 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612311 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900977 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913434 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720322 s, 14.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789302 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785568 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129738 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799059 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956916 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104023 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135184 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141181 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145806 s, 7.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.419407 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769008 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963943 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159698 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184421 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233568 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242944 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288983 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309981 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288364 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247173 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288602 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233918 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314581 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243594 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307602 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362245 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106459 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143069 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115969 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132054 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118065 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150601 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159255 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163752 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129975 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139662 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167281 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176932 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189981 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152836 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198728 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155281 s, 6.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=7 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.411458 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837047 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10345 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854031 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946328 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940921 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13971 s, 7.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194382 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238177 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247679 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356698 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330188 s, 3.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366341 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377847 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336011 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312351 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442595 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529874 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201494 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856333 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110972 s, 94.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180674 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300229 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548598 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326441 s, 32.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401632 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986177 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548912 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510394 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571685 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455009 s, 230 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609296 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148002 s, 70.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.409326 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830218 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942466 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134121 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137684 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125515 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181266 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196781 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167269 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131967 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136461 s, 7.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221599 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207333 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136945 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586815 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163408 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250637 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511145 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00556297 s, 188 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711042 s, 14.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00806093 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401268 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00983756 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102551 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134141 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00898584 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675799 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795354 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544605 s, 193 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743604 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852919 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03449 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077605 s, 13.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (21s) == sanityn test 77f: check TBF JobID nrs policy ========== 19:59:05 (1789603145) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.390115 s, 43.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748826 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100033 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667389 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152995 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177872 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188402 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192346 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176183 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222855 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186735 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210854 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221852 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282681 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218982 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216863 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319967 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749098 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708137 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395184 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594725 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592833 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107132 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740654 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113208 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390401 s, 26.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959822 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602905 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492642 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134985 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468032 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044549 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415663 s, 25.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.395103 s, 42.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083463 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897926 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152939 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20311 s, 5.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147238 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211979 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158413 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205756 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238225 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212878 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291015 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261749 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302965 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237611 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386047 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356418 s, 29.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212711 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560725 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466188 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372913 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929761 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873606 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459915 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613535 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551039 s, 19.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836936 s, 12.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194473 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578027 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916661 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933631 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00955554 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102389 s, 102 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.501958 s, 33.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960389 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698579 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190432 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170382 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239051 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214931 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262014 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204098 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266051 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204545 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133998 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266439 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156688 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204382 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954978 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189161 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419842 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638616 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452921 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574577 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617262 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062733 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107756 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861248 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823308 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814052 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107221 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100805 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116657 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10758 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109004 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109017 s, 9.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.39831 s, 42.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693822 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062329 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890657 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12285 s, 8.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120163 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272477 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194853 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216878 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290988 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953724 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118899 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630016 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149797 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931671 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172326 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345905 s, 3.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04577 s, 22.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245201 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288916 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791285 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730865 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695643 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049832 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580063 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291918 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694997 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129687 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309308 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387567 s, 27.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294841 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00605061 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343468 s, 3.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (21s) == sanityn test 77g: Change TBF type directly ============ 19:59:26 (1789603166) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (9s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 19:59:35 (1789603175) oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg633-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg633-client: oleg633-server: ssh exited with exit code 2 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg633-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg633-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg633-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 PASS 77h (10s) == sanityn test 77i: Change rank of TBF rule ============= 19:59:45 (1789603185) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (19s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 20:00:04 (1789603204) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.482902 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074704 s, 14.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758558 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100971 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10175 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150998 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145379 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108247 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18599 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18003 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173591 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212212 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17551 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22532 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225152 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168753 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200441 s, 5.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379113 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565094 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237128 s, 442 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00302979 s, 346 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0076941 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374203 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458436 s, 229 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00942947 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108984 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849243 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117702 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051985 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889681 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103389 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387807 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412461 s, 25.4 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.2814 s, 12.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6121 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (64s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 20:01:08 (1789603268) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg633-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg633-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg633-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg633-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27027 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591073 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661243 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123298 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123374 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120764 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15757 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139801 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218005 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232588 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240753 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262402 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259797 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257621 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289209 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273731 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24397 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698892 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662859 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703115 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080835 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822004 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716345 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736092 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618568 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871286 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821932 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771829 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965677 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815402 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794186 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724034 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977432 s, 10.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9777 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.952 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg633-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg633-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg633-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg633-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33737 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117477 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136529 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156824 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178831 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203692 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226416 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174351 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241167 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222174 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249313 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27778 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276832 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366581 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290409 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307549 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422271 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623087 s, 16.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580757 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659091 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678305 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517039 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474175 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613848 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777335 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746262 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636174 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781033 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838867 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691152 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696287 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700504 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088061 s, 11.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8493 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9124 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (151s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 20:03:39 (1789603419) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg633-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg633-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg633-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg633-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29021 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695195 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101525 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102026 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607119 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101883 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709558 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146901 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201044 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186362 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207291 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196955 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226225 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201668 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231448 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250691 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245162 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850233 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897285 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101513 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929758 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102792 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111069 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102132 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114443 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115935 s, 9.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123526 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116006 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130717 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12956 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118367 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134225 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136181 s, 7.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.851 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.969 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg633-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg633-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg633-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg633-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.35137 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108886 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564832 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129956 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187332 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162126 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203173 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237688 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279992 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26632 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28951 s, 3.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339864 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333148 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277041 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346137 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358375 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35619 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792225 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838196 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252966 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824151 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785268 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557037 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379823 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048156 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872269 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800822 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753227 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866297 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647359 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853965 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631765 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699502 s, 15.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0528 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9418 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (149s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 20:06:08 (1789603568) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.764172 s, 22.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109199 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149449 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126635 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158059 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168061 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156026 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155145 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229738 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269812 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987345 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144209 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251438 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182762 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191473 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317879 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123655 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450894 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402428 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580481 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802865 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483396 s, 21.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058983 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908491 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961967 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114408 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799794 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05092 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592119 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126862 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861866 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122897 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139524 s, 7.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.65791 s, 13.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91331 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.33@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.33@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.480829 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571388 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981259 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108819 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139808 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210752 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154746 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19254 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215077 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213994 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210033 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327839 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243141 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324798 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355382 s, 3.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305502 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336262 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749495 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318609 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561642 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042934 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499689 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579032 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841124 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727767 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104616 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777863 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107687 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097197 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103813 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101919 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200502 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201503 s, 5.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.39331 s, 11.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89541 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.33@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.787454 s, 21.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786584 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921979 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701065 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107857 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149986 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236533 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238195 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198956 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195475 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221263 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211993 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254141 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244302 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339806 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308662 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211976 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228917 s, 458 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687171 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060006 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129194 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814115 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708519 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365736 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181845 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806867 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114329 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00318543 s, 329 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106215 s, 98.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102653 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00481342 s, 218 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148373 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171413 s, 61.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.19959 s, 11.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.43885 s, 19.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.815324 s, 20.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849868 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982898 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165068 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102994 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161035 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189487 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184101 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240058 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205771 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288121 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284152 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251698 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244938 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315614 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356034 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333358 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637861 s, 16.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512971 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459513 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236302 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747439 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306577 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621241 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057586 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629594 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617314 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366104 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705967 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467559 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511162 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051159 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614315 s, 17.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4934 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9438 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.15572 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87257 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.04498 s, 16.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601064 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827766 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121605 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935165 s, 11.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191928 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177162 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133276 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167837 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149636 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138795 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153265 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163162 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197992 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241222 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905701 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15493 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819684 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066046 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759026 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953788 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512874 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103965 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086555 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792846 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842121 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799999 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815361 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789431 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967322 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898423 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845254 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963496 s, 10.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7434 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6098 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.6741 s, 24.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688761 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181627 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100844 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196184 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202856 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294561 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181294 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184841 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305124 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242239 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433311 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474158 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26561 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481008 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431779 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487208 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510154 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412885 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119635 s, 87.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759024 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628231 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136749 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141126 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122374 s, 85.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889349 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813896 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153401 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144993 s, 7.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106269 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902678 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901418 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00477195 s, 220 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.76098 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.911 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.776378 s, 21.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109983 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106636 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295926 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269221 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342695 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31867 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321893 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304613 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312435 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405681 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361465 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344179 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328972 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324423 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438557 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388018 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092609 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774943 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107167 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117482 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106725 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118769 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082849 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850909 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793624 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959615 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101428 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942122 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10648 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101026 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115289 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132632 s, 7.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2002 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0045 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8034 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0653 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (439s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 20:13:27 (1789604007) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.542246 s, 30.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609531 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967447 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644645 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707936 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911813 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400725 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344592 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163871 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14155 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161056 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283782 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31361 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285394 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332636 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208241 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345938 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884306 s, 11.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502641 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356791 s, 294 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121392 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092753 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795625 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442077 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707198 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988526 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099863 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106151 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372029 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851646 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428139 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038702 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363745 s, 28.8 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.80021 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93502 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (54s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 20:14:21 (1789604061) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.33@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.33@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.735658 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960099 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158114 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204203 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205275 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261843 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248407 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347286 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30188 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355504 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352217 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396186 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406055 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431743 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391767 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439014 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354203 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150869 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189372 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184456 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187728 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168676 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190706 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167498 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201537 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166067 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197925 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20234 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217802 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218721 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214092 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221304 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226535 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=3 removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.60091 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89771 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (51s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 20:15:12 (1789604112) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.33@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.582798 s, 28.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083072 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968763 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219903 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238179 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238721 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246603 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240192 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306004 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28874 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314715 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288087 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321323 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287093 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280075 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290705 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352202 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375769 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00648866 s, 162 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139112 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355116 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744154 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502013 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00797392 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486443 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871369 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585615 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049584 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00444048 s, 236 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00629534 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00332591 s, 315 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00893238 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126732 s, 82.7 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.63502 s, 13.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18291 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (43s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 20:15:55 (1789604155) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.638194 s, 26.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062528 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168091 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200777 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135906 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144021 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174786 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159163 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198475 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269434 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192283 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289093 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233266 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22737 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294244 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234537 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356164 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709371 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671005 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665504 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264783 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273751 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404698 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282167 s, 37.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807421 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828635 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049853 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105734 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504988 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526316 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056294 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574778 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058244 s, 18.0 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8677 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9596 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.62017 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96587 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (96s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 20:17:31 (1789604251) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.03975 s, 16.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134295 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179758 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256971 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286062 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305222 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121079 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296132 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312584 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374469 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446874 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456426 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466771 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398488 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237814 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348303 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324879 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377256 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584879 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047442 s, 22.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673163 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605715 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072989 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741938 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908971 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676032 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694393 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692788 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732094 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978204 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918958 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865366 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110061 s, 9.5 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8112 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9221 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (74s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 20:18:45 (1789604325) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.663858 s, 25.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850241 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110117 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662556 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873847 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588067 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624698 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123686 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197685 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1232 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252432 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198732 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267619 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166309 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271345 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285458 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283969 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695349 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872577 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764042 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100459 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822406 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894747 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113903 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117812 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849175 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137505 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142173 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108091 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814241 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107986 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931737 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011248 s, 93.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.48733 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87843 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg633-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.912672 s, 18.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325133 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993766 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107151 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900475 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122277 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12014 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192722 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207239 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164618 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189394 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268635 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163449 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240634 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283296 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287009 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33692 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371819 s, 282 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109171 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108198 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00223496 s, 469 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00591268 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126858 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00614062 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645561 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00815577 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00945989 s, 111 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961193 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127151 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160892 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164934 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651033 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13513 s, 7.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3341 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96959 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7977 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91734 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (147s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 20:21:12 (1789604472) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg633-client.virtnet /mnt/lustre (opts:) Stopping client oleg633-client.virtnet /mnt/lustre opts: Stopping clients: oleg633-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg633-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg633-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg633-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg633-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg633-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.33,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg633-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.33,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg633-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.33,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70372/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.33,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg633-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg633-server' oleg633-server: oleg633-server.virtnet: executing load_modules_local oleg633-server: kmemleak disabled oleg633-server: Loading modules from /home/green/git/lustre-release/lustre oleg633-server: detected 4 online CPUs by sysfs oleg633-server: MODOPTS_LIBCFS= oleg633-server: Force libcfs to create 2 CPU partitions oleg633-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg633-server: oleg633-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg633-client: oleg633-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre Starting client oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre Started clients oleg633-client.virtnet: 192.168.206.133@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 Starting client oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 Started clients oleg633-client.virtnet: 192.168.206.133@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff936444f00000.idle_timeout=debug osc.lustre-OST0000-osc-ffff936458d01000.idle_timeout=debug osc.lustre-OST0001-osc-ffff936444f00000.idle_timeout=debug osc.lustre-OST0001-osc-ffff936458d01000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.493879 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384622 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758013 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105929 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960853 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111649 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149509 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15075 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140596 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136868 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173264 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180896 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165943 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179189 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188424 s, 5.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215169 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229156 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719331 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066811 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823153 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299007 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584452 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208225 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595445 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996258 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569856 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664744 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613629 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497282 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561084 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00800487 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619216 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077144 s, 13.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9142 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23598 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.709078 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917987 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137465 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209702 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292458 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317775 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307241 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286215 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337167 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284786 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277722 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250983 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295076 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277248 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35079 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339535 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34537 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100504 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479176 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862551 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117874 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882577 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12089 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994063 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126175 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145512 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087011 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111599 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111409 s, 9.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102986 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10187 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098627 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163788 s, 6.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00000.lock_unused_count=1 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.0979 s, 11.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.925 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (222s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 20:24:54 (1789604694) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg633-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 20:25:12 (1789604712) On MGS 192.168.206.133, active = nodemap.active=1 On MGS 192.168.206.133, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.524217 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131648 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985409 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263119 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269777 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255791 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31911 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336082 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435321 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419594 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386814 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.489792 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426867 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45481 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.478712 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445046 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455403 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800294 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848848 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901881 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137272 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100599 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103376 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110916 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113238 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143725 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102791 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121597 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125679 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119555 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128798 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13077 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147059 s, 7.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2337 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.24101 s, 20.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.206.133, TBF.id = On MGS 192.168.206.133, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.640812 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037553 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721251 s, 14.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110323 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744925 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135262 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134187 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120844 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104882 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693104 s, 15.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936444f00000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274772 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370182 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316874 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374169 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108617 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143448 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152912 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090307 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124802 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898019 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139039 s, 7.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158868 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127081 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275953 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114888 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117885 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461682 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387499 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176892 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116717 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00924033 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376154 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402711 s, 26.0 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.38751 s, 14.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.83737 s, 37.0 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg633-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg633-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 On MGS 192.168.206.133, active = nodemap.active=1 On MGS 192.168.206.133, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.672297 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531317 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648275 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981208 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179386 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210582 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237253 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277718 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328951 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362003 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362032 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357671 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381785 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369982 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386726 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393117 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.492011 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303775 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285973 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206552 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290381 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245844 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230541 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264445 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243315 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292299 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271216 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294535 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325082 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280729 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37423 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328352 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370645 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00000.lock_unused_count=1 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8782 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92091 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.206.133, TBF.id = On MGS 192.168.206.133, active = nodemap.active=0 PASS 77kj (184s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 20:28:16 (1789604896) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 20:28:27 (1789604907) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg633-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.64899 s, 287 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg633-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.009 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (56s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 20:29:23 (1789604963) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.489866 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416327 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580658 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185901 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191579 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187956 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208734 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228164 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229084 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24154 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275805 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288007 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267729 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2627 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294823 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329138 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362571 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff936444f00000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842783 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828452 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440157 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587208 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765387 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632094 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876675 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843187 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807135 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533001 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961953 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785613 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695098 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093789 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662458 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101338 s, 10.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg633-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg633-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.92733 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09676 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.46994 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570904 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694234 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717394 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757055 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120646 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124712 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154922 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236072 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172421 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196078 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170109 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218823 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340197 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226859 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324914 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312637 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936444f00000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119788 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913817 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811335 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829415 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828717 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491966 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715729 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10193 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874982 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10183 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791969 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911159 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110942 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122887 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103314 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114093 s, 9.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg633-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg633-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.16178 s, 14.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17875 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (85s) == sanityn test 77o: Changing rank should not panic ====== 20:30:48 (1789605048) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (11s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 20:30:59 (1789605059) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (102s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 20:32:41 (1789605161) mds.MDS.mdt.nrs_policies=tbf oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg633-client: oleg633-server: ssh exited with exit code 36 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 oleg633-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg633-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg633-client: oleg633-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (35s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 20:33:16 (1789605196) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273701 kill md_thread xargs PASS 77r (53s) == sanityn test 77s: Check TBF LRU shrinker ============== 20:34:09 (1789605249) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (55s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 20:35:05 (1789605305) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 20:35:14 (1789605314) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935707 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 20:35:15.000000000 -0400 Modify: 2026-09-16 20:35:15.000000000 -0400 Change: 2026-09-16 20:35:15.000000000 -0400 Birth: 2026-09-16 20:35:14.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (18s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 20:35:31 (1789605331) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 20:35:34 (1789605334) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 20:35:37 (1789605337) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 20:35:40 (1789605340) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 20:35:43 (1789605343) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 20:35:46 (1789605346) - open/close 850 (time 1789605361.22 total 10.00 last 84.97) - open/close 1592 (time 1789605371.22 total 20.01 last 74.19) - open/close 2372 (time 1789605381.22 total 30.01 last 77.98) total: 2500 open/close in 31.52 seconds: 79.31 ops/second snapshot_time 1789605451.531661587 secs.nsecs start_time 1789605348.250790310 secs.nsecs elapsed_time 103.280871277 secs.nsecs open 2505 samples [usecs] 114 16667 3650906 9136981818 close 2504 samples [usecs] 133 10260 835558 586782712 mknod 2501 samples [usecs] 561 14753 3169229 7207991789 unlink 2500 samples [usecs] 675 92659 5047344 45964745920 mkdir 2 samples [usecs] 2136 4140 6276 21702096 rmdir 2 samples [usecs] 6981 14014 20995 245126557 rename 2500 samples [usecs] 434 34204 13662675 144124898487 getattr 3762 samples [usecs] 7 17633 245796 657761972 parallel_rename_file 2500 samples [usecs] 434 34204 13662675 144124898487 crossdir_rename 2500 samples [usecs] 434 34204 13662675 144124898487 readpage 24 samples [usecs] 3828 46412 434712 10586565348 PASS 81d (110s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 20:37:36 (1789605456) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7650 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7650 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 20:37:43 (1789605463) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 20:37:45 (1789605465) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 20:37:58 (1789605478) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789605479 Finishing test test1 at 1789605483 PASS 85 (9s) == sanityn test 90: open/create and unlink striped directory ========================================================== 20:38:07 (1789605487) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 20:38:10 (1789605490) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 20:38:13 (1789605493) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 20:38:16 (1789605496) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5886 0x16fe 0x240000400 1 5719 0x1657 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5887 0x16ff 0x240000400 1 5720 0x1658 0x280000400 rc1=0 and rc2=0 PASS 93 (15s) == sanityn test 94: signal vs CP callback race =========== 20:38:31 (1789605511) osc.lustre-OST0000-osc-ffff936444f00000.idle_timeout=0 osc.lustre-OST0000-osc-ffff936458d01000.idle_timeout=0 osc.lustre-OST0001-osc-ffff936444f00000.idle_timeout=0 osc.lustre-OST0001-osc-ffff936458d01000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.186909 s, 43.8 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00942 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290876 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.045449 s, 90.1 kB/s osc.lustre-OST0000-osc-ffff936444f00000.idle_timeout=debug osc.lustre-OST0000-osc-ffff936458d01000.idle_timeout=debug osc.lustre-OST0001-osc-ffff936444f00000.idle_timeout=debug osc.lustre-OST0001-osc-ffff936458d01000.idle_timeout=debug PASS 94 (14s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 20:38:45 (1789605525) llite.lustre-ffff936444f00000.fast_read=0 llite.lustre-ffff936458d01000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100913 s, 20.8 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff936444f00000.fast_read=1 llite.lustre-ffff936458d01000.fast_read=1 PASS 95a (19s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 20:39:04 (1789605544) llite.lustre-ffff936444f00000.fast_read=0 llite.lustre-ffff936458d01000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0074244 s, 1.7 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff936444f00000.fast_read=1 llite.lustre-ffff936458d01000.fast_read=1 PASS 95b (12s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 20:39:17 (1789605557) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 20:39:20 (1789605560) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00552164 s, 742 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 20:39:26 (1789605566) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.197405 s, 10.6 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 20:39:34 (1789605574) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118684 s, 17.7 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 20:39:42 (1789605582) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0145053 s, 827 kB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 20:39:49 (1789605589) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00872593 s, 469 kB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 20:39:55 (1789605595) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0122575 s, 334 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 20:40:02 (1789605602) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00966178 s, 424 kB/s PASS 101c (7s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 20:40:09 (1789605609) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 7c 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf7c:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (6s) == sanityn test 103: Test size correctness with lockahead ========================================================== 20:40:15 (1789605615) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789605617 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789605620 Finishing test test23 at 1789605620 PASS 103 (11s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 20:40:26 (1789605626) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 20:40:29 (1789605629) fail_loc=0x416 PASS 105 (22s) == sanityn test 106a: Verify the btime via statx() ======= 20:40:51 (1789605651) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 20:40:54 (1789605654) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00895134 s, 114 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938243 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 20:40:55.000000000 -0400 Modify: 2026-09-16 20:40:55.000000000 -0400 Change: 2026-09-16 20:40:55.000000000 -0400 Birth: 2026-09-16 20:40:54.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938243 -rw-r--r-- 1 0 0 1789605654 1789605655 1789605655 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938243 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 20:40:55.000000000 -0400 Modify: 2026-09-16 20:40:55.000000000 -0400 Change: 2026-09-16 20:40:55.000000000 -0400 Birth: 2026-09-16 20:40:54.000000000 -0400 1789605655 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 20:41:01 (1789605661) PASS 106c (6s) == sanityn test 107a: Basic grouplock conflict =========== 20:41:07 (1789605667) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.271778 s, 38.6 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5896 0x1708 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 20:41:16 (1789605676) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.362618 s, 28.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5727 0x165f 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7650 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 20:41:29 (1789605689) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00373488 s, 2.7 MB/s llite.lustre-ffff936444f00000.hybrid_io=0 llite.lustre-ffff936458d01000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.57344 s, 1.8 MB/s PASS 108a (11s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 20:41:40 (1789605700) 192.168.206.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg633-client.virtnet /mnt/lustre (opts:) 192.168.206.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg633-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre 192.168.206.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg633-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg633-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg633-server' oleg633-server: oleg633-server.virtnet: executing load_modules_local oleg633-server: kmemleak disabled oleg633-server: Loading modules from /home/green/git/lustre-release/lustre oleg633-server: detected 4 online CPUs by sysfs oleg633-server: MODOPTS_LIBCFS= oleg633-server: Force libcfs to create 2 CPU partitions oleg633-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 192.168.206.133@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg633-client.virtnet /mnt/lustre (opts:) Stopping client oleg633-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg633-server' oleg633-server: oleg633-server.virtnet: executing load_modules_local oleg633-server: kmemleak disabled oleg633-server: Loading modules from /home/green/git/lustre-release/lustre oleg633-server: detected 4 online CPUs by sysfs oleg633-server: MODOPTS_LIBCFS= oleg633-server: Force libcfs to create 2 CPU partitions oleg633-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre Starting client: oleg633-client.virtnet: -o user_xattr,flock 192.168.206.133@tcp:/lustre /mnt/lustre2 PASS 109 (61s) == sanityn test 110: do not grant another lock on resend ========================================================== 20:42:42 (1789605762) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 20:42:43.000000000 -0400 Modify: 2026-09-16 20:42:43.000000000 -0400 Change: 2026-09-16 20:42:43.000000000 -0400 Birth: 2026-09-16 20:42:42.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg633-client.virtnet oleg633-client.virtnet PASS 110 (168s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 20:45:30 (1789605930) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 20:45:34 (1789605934) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 20:45:37 (1789605937) lustre-OST0000-osc-ffff9364431ee000 active. lustre-OST0001-osc-ffff9364431ee000 active. lustre-OST0000-osc-ffff9364431ee000 active. lustre-OST0001-osc-ffff9364431ee000 active. MGC192.168.206.133@tcp active. PASS 113 (5s) == sanityn test 114: implicit default LMV inherit ======== 20:45:42 (1789605942) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 20:45:45 (1789605945) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (2s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 20:45:47 (1789605947) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 20:45:50 (1789605950) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00830113 s, 2.5 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.058718 s, 349 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (8s) == sanityn test 122: directory size is consistent across mounts ========================================================== 20:45:58 (1789605958) PASS 122 (7s) == sanityn test 200: service remains healthy while able to process request ========================================================== 20:46:05 (1789605965) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=11 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0798 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1721 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1243 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1459 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1416 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1274 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1428 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1599 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1763 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1121 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1983 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0748 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1838 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.178 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2137 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2185 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2793 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2178 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2336 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3154 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2928 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3396 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg633-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9364431ee000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9364431ee000.ost_server_uuid in FULL state after 0 sec PASS 200 (80s) cleanup: ====================================================== == sanityn test complete, duration 9731 sec ============== 20:47:27 (1789606047) === sanityn: start cleanup 20:47:29 (1789606049) === Stopping clients: oleg633-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg633-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 20:51:11 (1789606271) ===