-----============= acceptance-small: sanityn ============----- Sat Sep 5 17:01:46 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg411-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 17:02:04 (1788642124) === Starting client oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 Mount client oleg411-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 Started clients oleg411-client.virtnet: 192.168.204.111@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg411-client.virtnet: executing check_config_client /mnt/lustre oleg411-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg411-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e9fc58a7800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e9fc6e7a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e9fc58a7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e9fc6e7a800.idle_timeout=debug disable quota as required oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 17:02:34 (1788642154) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215713 s, 48.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7655] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 17:02:36 (1788642156) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 17:02:43 (1788642163) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 17:02:49 (1788642169) -rw-r--r-- 1 root root 0 Sep 5 17:02 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 17:02:56 (1788642176) -rw-r--r-- 1 root root 0 Sep 5 17:02 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 17:03:02 (1788642182) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 17:03:08 (1788642188) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 17:03:15 (1788642195) total 1 -rwxrwxrwx 1 root root 0 Sep 5 17:02 f2a -rwxrwxrwx 1 root root 0 Sep 5 17:02 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 5 17:02 f2a -rwxrwxrwx 1 root root 0 Sep 5 17:02 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00871422 s, 58.8 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 17:03:21 (1788642201) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 17:03:24 (1788642204) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.266372 s, 7.9 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 17:03:30 (1788642210) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 17:03:35 (1788642215) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 17:03:43 (1788642223) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 17:03:50 (1788642230) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 17:03:57 (1788642237) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 17:04:03 (1788642243) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 17:04:09 (1788642249) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 17:04:16 (1788642256) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554151 s, 18.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0320518 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 17:04:23 (1788642263) 1+0 records in 1+0 records out 1 byte copied, 0.00846918 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00722919 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00682412 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00649142 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0184161 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00598633 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00865693 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00815253 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0218933 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011401 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00909624 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00771749 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 17:04:31 (1788642271) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0254354 s, 121 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0254348 s, 161 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000363579 s, 8.4 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 17:04:38 (1788642278) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 17:04:45 (1788642285) warning: '-runtime' deprecated, use '-t runtime' instead - link 370 (time 1788642297.53 total 10.01 last 36.97) - link 724 (time 1788642307.54 total 20.02 last 35.36) - link 1058 (time 1788642317.54 total 30.03 last 33.38) - link 1388 (time 1788642327.56 total 40.04 last 32.94) - link 1738 (time 1788642337.59 total 50.07 last 34.92) - link 2074 (time 1788642347.60 total 60.08 last 33.54) - link 2436 (time 1788642357.61 total 70.10 last 36.15) - link 2814 (time 1788642367.64 total 80.12 last 37.70) - link 3197 (time 1788642377.67 total 90.15 last 38.20) total: 3500 link in 100.04 seconds: 34.99 ops/second using seed 3404113683 running for 100 seconds - stat 4945 (time 1788642299 ; total 11 ; last 11) - stat 10000 (time 1788642308 ; total 20 ; last 9) - stat 15136 (time 1788642319 ; total 31 ; last 11) - stat 20000 (time 1788642328 ; total 40 ; last 9) - stat 25516 (time 1788642339 ; total 51 ; last 11) - stat 28471 (time 1788642350 ; total 62 ; last 11) - stat 30000 (time 1788642355 ; total 67 ; last 5) - stat 32827 (time 1788642366 ; total 78 ; last 11) - stat 35526 (time 1788642377 ; total 89 ; last 11) - stat 38797 (time 1788642388 ; total 100 ; last 11) total: 38798 stats in 100 seconds: 387.980011 stats/second - unlinked 0 (time 1788642338 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3500) error: No such file or directory total: 3499 unlinks in 91 seconds: 38.450550 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21456) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21458) - No such process PASS 12 (149s) == sanityn test 13: test directory page revocation ======= 17:07:14 (1788642434) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 17:07:23 (1788642443) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22641 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 17:07:29 (1788642449) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23220 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 17:07:36 (1788642456) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23799 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 17:07:42 (1788642462) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24381 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 17:07:49 (1788642469) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24963 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 17:07:59 (1788642479) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (53s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 17:08:51 (1788642531) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.367997 s, 22.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.464977 s, 18.0 MB/s Chance of close/open is 1 in 50 Seed set to 2536 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x466ffb 000001[1] 1788642536.919655 trunc from 00000000 to 0x466ffa (0x466ffb bytes) 000002[1] 1788642536.939845 write 0x4ecfaa thru 0x4ee239 (0x1290 bytes) 000003[0] 1788642536.940901 read 0x348508 thru 0x34bfc0 (0x3ab9 bytes) 000004[0] 1788642536.965940 write 0x7aae66 thru 0x7af212 (0x43ad bytes) 000005[0] 1788642536.989904 write 0x0e95c1 thru 0x0f91d0 (0xfc10 bytes) truncating to largest ever: 0x901827 000006[1] 1788642537.008702 trunc from 0x7af213 to 0x901826 (0x152614 bytes) 000009[0] 1788642537.142474 mapread 0x258e36 thru 0x260361 (0x752c bytes) 000010[1] 1788642537.222387 trunc from 0x901827 to 0x4b1560 (0x4502c6 bytes) 000012[1] 1788642537.254290 trunc from 0x4b1561 to 0x1d10c4 (0x2e049c bytes) 000014[0] 1788642537.277489 punch from 0x0dfa48 to 0x0e7909 (0x7ec2 bytes) 000015[1] 1788642537.306722 read 0x0ebeb0 thru 0x0f3aea (0x7c3b bytes) 000017[0] 1788642537.365715 trunc from 0x1d10c5 to 0x01cb65 (0x1b455f bytes) 000018[0] 1788642537.441029 mapread 0x01a76e thru 0x01cb65 (0x23f8 bytes) 000019[0] 1788642537.495064 mapwrite 0x724843 thru 0x726a6a (0x2228 bytes) 000021[1] 1788642537.813904 trunc from 0x726a6b to 0x5b2c0c (0x173e5e bytes) 000023[1] 1788642537.897357 trunc from 0x5b2c0d to 0x7f2fb2 (0x2403a6 bytes) 000024[1] 1788642537.936599 read 0x6701eb thru 0x67d07e (0xce94 bytes) 000025[1] 1788642537.987736 read 0x74c2eb thru 0x7542d6 (0x7fec bytes) 000026[0] 1788642537.996475 read 0x1e3f41 thru 0x1ed033 (0x90f3 bytes) 000028[0] 1788642538.062977 write 0x2bdf48 thru 0x2c9aa8 (0xbb61 bytes) 000029[0] 1788642538.096337 read 0x19d171 thru 0x1a6d52 (0x9be2 bytes) 000031[0] 1788642538.136898 punch from 0x2f76e2 to 0x2f9ec0 (0x27df bytes) 000032[0] 1788642538.167370 read 0x51aeaf thru 0x5297ce (0xe920 bytes) 000034[0] 1788642538.207054 punch from 0x4a578b to 0x4b4025 (0xe89b bytes) 000035[1] 1788642538.252543 read 0x019878 thru 0x02309c (0x9825 bytes) 000036[0] 1788642538.290893 write 0x198383 thru 0x19cc52 (0x48d0 bytes) 000038[0] 1788642538.384065 read 0x263504 thru 0x2731d2 (0xfccf bytes) 000039[1] 1788642538.403084 read 0x40e6c0 thru 0x41283f (0x4180 bytes) 000040[1] 1788642538.424668 read 0x0e997d thru 0x0f0117 (0x679b bytes) truncating to largest ever: 0x9609d2 000041[0] 1788642538.457284 trunc from 0x7f2fb3 to 0x9609d1 (0x16da1f bytes) 000045[1] 1788642538.622995 read 0x2d03ea thru 0x2db89d (0xb4b4 bytes) 000046[1] 1788642538.672672 trunc from 0x9609d2 to 0x603b85 (0x35ce4c bytes) 000047[1] 1788642538.768512 read 0x5c4aaa thru 0x5c8e9d (0x43f4 bytes) 000048[1] 1788642538.792521 mapread 0x1a03b9 thru 0x1afa5e (0xf6a6 bytes) 000051[0] 1788642538.884600 mapread 0x2a1192 thru 0x2ae70c (0xd57b bytes) 000052[1] 1788642538.951321 write 0x703dcd thru 0x704b3e (0xd72 bytes) 000053[1] 1788642538.971838 trunc from 0x704b3f to 0x1611ac (0x5a3992 bytes) 000054[0] 1788642539.018846 mapwrite 0x866a33 thru 0x869f1b (0x34e9 bytes) 000057[1] 1788642539.154826 punch from 0x5bc877 to 0x5c4d6a (0x84f4 bytes) 000058[0] 1788642539.184328 mapread 0x74b9b8 thru 0x75694c (0xaf95 bytes) 000059[0] 1788642539.240572 mapwrite 0x0a3e16 thru 0x0b388b (0xfa76 bytes) 000060[1] 1788642539.428884 punch from 0x702cae to 0x70df38 (0xb28b bytes) 000061[0] 1788642539.472479 trunc from 0x869f1c to 0x225353 (0x644bc8 bytes) 000062[0] 1788642539.508648 write 0x8ca89d thru 0x8d5542 (0xaca6 bytes) 000063[1] 1788642539.558478 write 0x3d66f0 thru 0x3dfc38 (0x9549 bytes) 000065[1] 1788642539.634446 punch from 0x5f3c7a to 0x5fc409 (0x8790 bytes) 000066[0] 1788642539.650541 read 0x60cf8d thru 0x611f02 (0x4f76 bytes) 000067[0] 1788642539.696458 write 0x68cd28 thru 0x68ff6f (0x3248 bytes) 000068[0] 1788642539.724518 trunc from 0x8d5543 to 0x0e5638 (0x7eff0a bytes) 000070[0] 1788642539.849608 mapread 0x0ab913 thru 0x0b227b (0x6969 bytes) 000071[1] 1788642539.885383 punch from 0x05a4dd to 0x0612a2 (0x6dc6 bytes) 000073[1] 1788642539.934624 write 0x6939d5 thru 0x6a1060 (0xd68c bytes) 000074[1] 1788642539.975025 mapread 0x2b49d9 thru 0x2beaf2 (0xa11a bytes) 000075[0] 1788642540.049766 read 0x5387e5 thru 0x53d4f4 (0x4d10 bytes) 000078[0] 1788642540.130713 write 0x88b8d9 thru 0x893b27 (0x824f bytes) 000079[0] 1788642540.177729 mapread 0x109471 thru 0x115280 (0xbe10 bytes) 000080[0] 1788642540.218355 mapread 0x0802ad thru 0x08d271 (0xcfc5 bytes) 000081[1] 1788642540.255184 punch from 0x761544 to 0x768880 (0x733d bytes) 000084[1] 1788642540.322017 punch from 0x3d0c4e to 0x3d672e (0x5ae1 bytes) 000086[1] 1788642540.344798 trunc from 0x893b28 to 0x5de3ab (0x2b577c bytes) 000087[1] 1788642540.393471 mapread 0x181376 thru 0x183e2a (0x2ab5 bytes) 000089[1] 1788642540.446612 mapwrite 0x79d0c0 thru 0x7ab569 (0xe4aa bytes) 000091[1] 1788642540.617722 trunc from 0x7ab56a to 0x1f8d39 (0x5b2830 bytes) 000092[1] 1788642540.645158 mapread 0x0273b7 thru 0x02f78e (0x83d8 bytes) 000093[0] 1788642540.674594 punch from 0x064349 to 0x06aa96 (0x674e bytes) 000094[0] 1788642540.706525 punch from 0x0e96d0 to 0x0f78c5 (0xe1f6 bytes) 000095[1] 1788642540.724852 trunc from 0x1f8d3a to 0x09d756 (0x15b5e3 bytes) 000099[1] 1788642540.840746 trunc from 0x09d757 to 0x24322a (0x1a5ad4 bytes) 000101[0] 1788642540.863165 mapwrite 0x5248dc thru 0x526b48 (0x226d bytes) 000102[0] 1788642541.016523 mapread 0x321f8c thru 0x329a23 (0x7a98 bytes) 000103[1] 1788642541.057000 read 0x3e83f8 thru 0x3e8b95 (0x79e bytes) 000105[0] 1788642541.097070 mapwrite 0x400ff1 thru 0x405868 (0x4878 bytes) 000107[0] 1788642541.210549 write 0x35c9d6 thru 0x3681cd (0xb7f8 bytes) 000108[0] 1788642541.240764 mapread 0x1485d0 thru 0x14d11b (0x4b4c bytes) 000109[1] 1788642541.286519 punch from 0x1de55f to 0x1df0d7 (0xb79 bytes) 000110[0] 1788642541.345503 punch from 0x057d0e to 0x05d942 (0x5c35 bytes) 000111[1] 1788642541.381666 read 0x3fae27 thru 0x402787 (0x7961 bytes) 000114[0] 1788642541.454114 mapread 0x481b3d thru 0x48b4cc (0x9990 bytes) 000115[0] 1788642541.494829 read 0x2c950c thru 0x2d93f5 (0xfeea bytes) 000117[1] 1788642541.520142 write 0x538e60 thru 0x547f29 (0xf0ca bytes) 000118[1] 1788642541.548640 write 0x5ba9d2 thru 0x5bc301 (0x1930 bytes) 000119[0] 1788642541.554155 punch from 0x18fa25 to 0x196703 (0x6cdf bytes) 000121[1] 1788642541.624670 punch from 0x0f0fb3 to 0x0f3504 (0x2552 bytes) 000122[1] 1788642541.687520 mapread 0x330207 thru 0x330564 (0x35e bytes) 000123[1] 1788642541.718630 punch from 0x074d94 to 0x079372 (0x45df bytes) 000125[1] 1788642541.741835 trunc from 0x5bc302 to 0x2832be (0x339043 bytes) 000126[1] 1788642541.789755 mapread 0x24ebfa thru 0x257437 (0x883e bytes) 000129[0] 1788642541.838804 write 0x326367 thru 0x3307cc (0xa466 bytes) 000130[1] 1788642541.870155 mapread 0x2b972c thru 0x2bf485 (0x5d5a bytes) 000134[1] 1788642541.974542 trunc from 0x3307cd to 0x7ff38b (0x4cebbf bytes) 000135[0] 1788642542.041748 read 0x611724 thru 0x618973 (0x7250 bytes) 000136[1] 1788642542.071770 mapread 0x06e0eb thru 0x071d4b (0x3c61 bytes) 000137[1] 1788642542.108602 write 0x071efa thru 0x07b389 (0x9490 bytes) 000138[0] 1788642542.109819 write 0x6a96aa thru 0x6aa95c (0x12b3 bytes) 000139[0] 1788642542.147779 trunc from 0x7ff38c to 0x134768 (0x6cac23 bytes) 000141[0] 1788642542.231559 punch from 0x060825 to 0x06d61a (0xcdf6 bytes) 000143[1] 1788642542.267551 read 0x08b7a8 thru 0x08e9b9 (0x3212 bytes) truncating to largest ever: 0x976145 000148[1] 1788642542.330457 trunc from 0x134769 to 0x976144 (0x8419dc bytes) 000149[1] 1788642542.418530 read 0x65388e thru 0x65a182 (0x68f5 bytes) 000151[0] 1788642542.478444 read 0x5f7c01 thru 0x5fcbac (0x4fac bytes) 000152[0] 1788642542.522969 read 0x3bc1da thru 0x3c58d0 (0x96f7 bytes) 000155[0] 1788642542.542518 read 0x5b6a92 thru 0x5c0467 (0x99d6 bytes) 000157[0] 1788642542.557559 mapread 0x0ef1db thru 0x0f707a (0x7ea0 bytes) 000161[1] 1788642542.575498 read 0x7df50c thru 0x7e6b69 (0x765e bytes) 000162[1] 1788642542.584845 trunc from 0x976145 to 0x6175da (0x35eb6a bytes) 000163[0] 1788642542.652747 trunc from 0x6175db to 0x70d7c6 (0xf61ec bytes) 000164[0] 1788642542.721818 mapread 0x49e50c thru 0x4a6697 (0x818c bytes) 000165[0] 1788642542.762882 punch from 0x1faf36 to 0x1ff42a (0x44f5 bytes) 000166[0] 1788642542.771933 punch from 0x466e69 to 0x46d168 (0x6300 bytes) 000167[0] 1788642542.803077 write 0x38dfb8 thru 0x38fb18 (0x1b61 bytes) 000168[0] 1788642542.814107 mapread 0x0e0064 thru 0x0e8055 (0x7ff2 bytes) 000170[0] 1788642542.865597 write 0x0abad4 thru 0x0b013c (0x4669 bytes) 000171[0] 1788642542.892159 mapread 0x5cb7ea thru 0x5d3485 (0x7c9c bytes) 000173[0] 1788642542.924386 punch from 0x5ea5f3 to 0x5ec125 (0x1b33 bytes) 000175[1] 1788642542.949462 punch from 0x5e73dc to 0x5f0578 (0x919d bytes) 000179[1] 1788642543.029837 read 0x36d210 thru 0x37a4fc (0xd2ed bytes) 000180[1] 1788642543.060317 read 0x5bd565 thru 0x5c2857 (0x52f3 bytes) 000182[0] 1788642543.089868 trunc from 0x70d7c7 to 0x42c549 (0x2e127d bytes) 000185[0] 1788642543.203939 read 0x3549e0 thru 0x35b5f2 (0x6c13 bytes) 000186[0] 1788642543.221097 trunc from 0x42c54a to 0x3920bf (0x9a48a bytes) 000188[1] 1788642543.288021 trunc from 0x3920c0 to 0x18582d (0x20c892 bytes) 000192[0] 1788642543.386197 write 0x5dde33 thru 0x5de11a (0x2e8 bytes) 000194[0] 1788642543.428486 write 0x5fcd83 thru 0x60aad9 (0xdd57 bytes) 000197[0] 1788642543.481707 punch from 0x4b32d4 to 0x4b93f9 (0x6126 bytes) 000199[1] 1788642543.515780 punch from 0x53fe5e to 0x54d93b (0xdade bytes) 000201[0] 1788642543.550104 read 0x1efc19 thru 0x1f8634 (0x8a1c bytes) 000202[1] 1788642543.640045 mapwrite 0x97a82d thru 0x980e2c (0x6600 bytes) 000205[0] 1788642543.846732 mapwrite 0x2de8e7 thru 0x2df39a (0xab4 bytes) 000206[1] 1788642543.938568 punch from 0x0c442f to 0x0d13d8 (0xcfaa bytes) 000207[1] 1788642543.960972 punch from 0x5da4b5 to 0x5e00ef (0x5c3b bytes) 000208[1] 1788642543.970945 read 0x7171a7 thru 0x71f7b5 (0x860f bytes) 000209[1] 1788642543.982512 mapwrite 0x19adc9 thru 0x1a9bfa (0xee32 bytes) 000211[1] 1788642544.149717 mapread 0x140138 thru 0x145148 (0x5011 bytes) 000212[0] 1788642544.163589 punch from 0x226075 to 0x22e883 (0x880f bytes) 000213[1] 1788642544.196950 mapwrite 0x215c4c thru 0x216e50 (0x1205 bytes) 000214[1] 1788642544.293609 trunc from 0x980e2d to 0x7183eb (0x268a41 bytes) 000215[0] 1788642544.339514 mapwrite 0x2135ed thru 0x217538 (0x3f4c bytes) 000217[0] 1788642544.486872 mapwrite 0x442c6c thru 0x448362 (0x56f7 bytes) 000218[0] 1788642544.593841 trunc from 0x7183ec to 0x014f4f (0x70349c bytes) 000219[1] 1788642544.652401 read 0x00ee79 thru 0x01455f (0x56e7 bytes) 000220[0] 1788642544.688998 write 0x9c8a45 thru 0x9d1574 (0x8b30 bytes) 000224[1] 1788642544.764874 mapwrite 0x2c5a84 thru 0x2cc710 (0x6c8d bytes) 000226[1] 1788642545.035561 read 0x21937a thru 0x21ef6a (0x5bf1 bytes) 000227[1] 1788642545.055759 write 0x145f77 thru 0x151355 (0xb3df bytes) truncating to largest ever: 0x9fa394 000228[1] 1788642545.084980 trunc from 0x9d1575 to 0x9fa393 (0x28e1f bytes) 000229[1] 1788642545.124115 mapwrite 0x4cfeac thru 0x4d0739 (0x88e bytes) 000230[1] 1788642545.174232 write 0x729faf thru 0x72e0c1 (0x4113 bytes) 000232[0] 1788642545.191560 mapwrite 0x5674f5 thru 0x574cfd (0xd809 bytes) 000234[1] 1788642545.506587 trunc from 0x9fa394 to 0x161baf (0x8987e4 bytes) 000235[0] 1788642545.642038 read 0x0a2938 thru 0x0a2c65 (0x32e bytes) 000236[1] 1788642545.675323 write 0x4f2bfe thru 0x500d06 (0xe109 bytes) 000237[0] 1788642545.700045 read 0x4b294e thru 0x4bb494 (0x8b47 bytes) 000239[1] 1788642545.734061 mapread 0x3c9705 thru 0x3ca60e (0xf0a bytes) 000240[0] 1788642545.742069 mapread 0x3ae9be thru 0x3be3e3 (0xfa26 bytes) 000241[1] 1788642545.817739 mapread 0x2f9b01 thru 0x3095f4 (0xfaf4 bytes) 000243[0] 1788642545.886580 mapread 0x0b31b4 thru 0x0b584d (0x269a bytes) 000246[1] 1788642545.907864 read 0x05ea31 thru 0x0644fe (0x5ace bytes) 000247[1] 1788642545.915460 write 0x5bdb41 thru 0x5ca438 (0xc8f8 bytes) 000248[0] 1788642545.946011 write 0x39109d thru 0x3910bf (0x23 bytes) 000250[0] 1788642546.002485 trunc from 0x5ca439 to 0x4851c4 (0x145274 bytes) 000252[0] 1788642546.122907 trunc from 0x4851c5 to 0x6d8b19 (0x253955 bytes) 000253[1] 1788642546.163425 read 0x30f1da thru 0x319fcd (0xadf4 bytes) 000255[1] 1788642546.218540 trunc from 0x6d8b1a to 0x51033b (0x1c87de bytes) 000256[1] 1788642546.259414 write 0x03edbc thru 0x04c4ce (0xd713 bytes) 000257[0] 1788642546.308969 mapwrite 0x662038 thru 0x671fef (0xffb8 bytes) 000259[1] 1788642546.426826 write 0x68d494 thru 0x696110 (0x8c7d bytes) 000260[0] 1788642546.451695 mapread 0x1fb19b thru 0x201144 (0x5faa bytes) 000263[0] 1788642546.515597 write 0x47ea6d thru 0x481075 (0x2609 bytes) 000264[1] 1788642546.559790 trunc from 0x696111 to 0x8be6c3 (0x2285b3 bytes) 000265[0] 1788642546.655471 read 0x6316ed thru 0x637e65 (0x6779 bytes) 000266[1] 1788642546.735317 punch from 0x5b4564 to 0x5b7a6b (0x3508 bytes) 000268[1] 1788642546.823811 punch from 0x8565e7 to 0x85cc4b (0x6665 bytes) 000272[0] 1788642546.963311 mapwrite 0x9226f8 thru 0x92f505 (0xce0e bytes) 000273[0] 1788642547.079340 mapread 0x5083c0 thru 0x50d9f6 (0x5637 bytes) 000274[1] 1788642547.124041 mapwrite 0x9bc0c9 thru 0x9c1b18 (0x5a50 bytes) 000276[0] 1788642547.298439 mapwrite 0x88285c thru 0x8845f2 (0x1d97 bytes) 000277[0] 1788642547.436958 mapread 0x6d9aaa thru 0x6e8f4b (0xf4a2 bytes) 000278[0] 1788642547.525818 mapread 0x33bb60 thru 0x349db0 (0xe251 bytes) 000279[0] 1788642547.560743 read 0x0b7039 thru 0x0b79a0 (0x968 bytes) 000281[1] 1788642547.578291 read 0x83c223 thru 0x842212 (0x5ff0 bytes) 000282[1] 1788642547.600605 mapwrite 0x71799a thru 0x718e89 (0x14f0 bytes) 000284[0] 1788642547.723205 mapwrite 0x69453c thru 0x6a1c6c (0xd731 bytes) 000285[0] 1788642547.850135 read 0x6ef972 thru 0x6ef9ec (0x7b bytes) 000287[1] 1788642547.857455 punch from 0x85815d to 0x85ff8d (0x7e31 bytes) 000288[0] 1788642547.903542 write 0x1cfd22 thru 0x1d8554 (0x8833 bytes) 000290[0] 1788642547.961322 write 0x6592ad thru 0x6637fe (0xa552 bytes) 000291[0] 1788642547.999242 write 0x1aba2a thru 0x1b8f96 (0xd56d bytes) 000293[1] 1788642548.039432 punch from 0x817a12 to 0x81b697 (0x3c86 bytes) 000294[0] 1788642548.078871 trunc from 0x9c1b19 to 0x4ac965 (0x5151b3 bytes) 000296[1] 1788642548.144415 trunc from 0x4ac966 to 0x94f8a1 (0x4a2f3c bytes) 000298[1] 1788642548.261845 trunc from 0x94f8a2 to 0x7bb1fc (0x1946a5 bytes) 000299[0] 1788642548.302425 mapwrite 0x242c2e thru 0x2444f4 (0x18c7 bytes) 000300[1] 1788642548.451591 trunc from 0x7bb1fd to 0x76d5ef (0x4dc0d bytes) 000302[1] 1788642548.508443 punch from 0x00ac42 to 0x01a48d (0xf84c bytes) 000304[1] 1788642548.604897 punch from 0x23309f to 0x239271 (0x61d3 bytes) 000305[0] 1788642548.634950 mapwrite 0x865a80 thru 0x86699f (0xf20 bytes) 000306[0] 1788642548.735550 write 0x95425b thru 0x960362 (0xc108 bytes) 000307[1] 1788642548.738648 write 0x8fea67 thru 0x90aa2a (0xbfc4 bytes) 000311[1] 1788642548.845044 trunc from 0x960363 to 0x2f8dab (0x6675b7 bytes) 000313[1] 1788642548.902397 punch from 0x2bf841 to 0x2c561f (0x5ddf bytes) 000314[1] 1788642548.911096 mapwrite 0x12c5e1 thru 0x138611 (0xc031 bytes) 000315[1] 1788642549.044849 mapwrite 0x8ac258 thru 0x8aff97 (0x3d40 bytes) 000316[1] 1788642549.157556 trunc from 0x8aff98 to 0x64c505 (0x263a92 bytes) 000317[1] 1788642549.181279 punch from 0x171667 to 0x175559 (0x3ef3 bytes) 000318[0] 1788642549.251830 mapwrite 0x2e9210 thru 0x2f1e71 (0x8c62 bytes) 000323[1] 1788642549.471988 punch from 0x0e6dcd to 0x0f42c7 (0xd4fb bytes) 000324[0] 1788642549.493086 read 0x3cc016 thru 0x3dbec4 (0xfeaf bytes) 000325[0] 1788642549.516534 mapread 0x414675 thru 0x41e9a3 (0xa32f bytes) 000328[1] 1788642549.608949 mapread 0x4fe3b9 thru 0x507345 (0x8f8d bytes) 000329[1] 1788642549.632076 trunc from 0x64c506 to 0x9f1bfd (0x3a56f8 bytes) 000332[1] 1788642549.759859 trunc from 0x9f1bfe to 0x422f9d (0x5cec60 bytes) 000333[0] 1788642549.800935 mapread 0x22071d thru 0x22eabf (0xe3a3 bytes) 000335[0] 1788642549.905647 mapwrite 0x265d10 thru 0x274cb4 (0xefa5 bytes) 000341[0] 1788642550.146675 mapwrite 0x4f3756 thru 0x4fa28a (0x6b35 bytes) 000342[0] 1788642550.269464 read 0x2f6e64 thru 0x3013c3 (0xa560 bytes) 000347[0] 1788642550.365613 read 0x200cc9 thru 0x208a44 (0x7d7c bytes) 000348[0] 1788642550.407058 trunc from 0x4fa28b to 0x98d39f (0x493115 bytes) 000350[1] 1788642550.445892 read 0x065679 thru 0x0691a3 (0x3b2b bytes) 000353[0] 1788642550.489097 mapread 0x574847 thru 0x57820e (0x39c8 bytes) 000355[1] 1788642550.564498 mapwrite 0x8eb723 thru 0x8fadcc (0xf6aa bytes) 000356[1] 1788642550.712520 mapwrite 0x64a4c1 thru 0x64e0bc (0x3bfc bytes) 000359[1] 1788642551.023173 trunc from 0x98d3a0 to 0x15181a (0x83bb85 bytes) 000360[0] 1788642551.078060 read 0x0fda95 thru 0x101a1b (0x3f87 bytes) 000361[0] 1788642551.132378 write 0x36c166 thru 0x37a7bb (0xe656 bytes) 000364[0] 1788642551.199757 mapread 0x16bf8f thru 0x1722bd (0x632f bytes) 000365[0] 1788642551.255447 trunc from 0x37a7bc to 0x474463 (0xf9ca8 bytes) 000366[1] 1788642551.285397 mapread 0x2661c8 thru 0x26cc3c (0x6a75 bytes) 000367[0] 1788642551.359632 write 0x6e1d93 thru 0x6ef048 (0xd2b6 bytes) 000369[1] 1788642551.367443 mapwrite 0x40eb91 thru 0x410c9d (0x210d bytes) 000370[1] 1788642551.465266 write 0x34932a thru 0x34b68a (0x2361 bytes) 000374[1] 1788642551.505513 mapread 0x2f5b06 thru 0x2fd31f (0x781a bytes) 000376[0] 1788642551.568616 mapwrite 0x3d715c thru 0x3da54d (0x33f2 bytes) 000377[1] 1788642551.689611 mapwrite 0x17dea4 thru 0x188d44 (0xaea1 bytes) 000379[0] 1788642551.841102 punch from 0x25cb50 to 0x260182 (0x3633 bytes) 000382[1] 1788642551.896029 read 0x2abb22 thru 0x2b56bf (0x9b9e bytes) 000383[1] 1788642551.934384 trunc from 0x6ef049 to 0x02d88a (0x6c17be bytes) 000385[1] 1788642551.993043 punch from 0x01fd8a to 0x0279c2 (0x7c39 bytes) 000386[1] 1788642551.997465 trunc from 0x02d88b to 0x9a56de (0x977e54 bytes) 000387[0] 1788642552.038898 read 0x15d29e thru 0x16cf24 (0xfc87 bytes) 000389[0] 1788642552.082511 read 0x82931e thru 0x8338f3 (0xa5d6 bytes) 000390[0] 1788642552.104037 punch from 0x0a05d8 to 0x0a4fa9 (0x49d2 bytes) 000394[0] 1788642552.208561 write 0x6feb6b thru 0x7022cc (0x3762 bytes) 000395[1] 1788642552.245037 trunc from 0x9a56df to 0x67f0cf (0x32660f bytes) 000399[1] 1788642552.379391 punch from 0x319de9 to 0x323d12 (0x9f2a bytes) 000400[0] 1788642552.409779 write 0x4c6d9c thru 0x4d0962 (0x9bc7 bytes) 000401[0] 1788642552.440293 trunc from 0x67f0d0 to 0x86c50d (0x1ed43e bytes) 000402[0] 1788642552.495404 mapread 0x5a4646 thru 0x5ad22b (0x8be6 bytes) 000405[0] 1788642552.520931 read 0x40f281 thru 0x4109fd (0x177d bytes) 000407[0] 1788642552.565117 mapread 0x00f076 thru 0x01b3cc (0xc357 bytes) 000412[0] 1788642552.681911 read 0x224d51 thru 0x22fb62 (0xae12 bytes) 000415[0] 1788642552.708397 mapwrite 0x76d1e1 thru 0x76e5cc (0x13ec bytes) 000417[0] 1788642552.790418 punch from 0x393e4e to 0x394231 (0x3e4 bytes) 000419[1] 1788642552.796029 write 0x0fef51 thru 0x109deb (0xae9b bytes) 000420[0] 1788642552.831332 mapwrite 0x48efa7 thru 0x499eea (0xaf44 bytes) 000423[1] 1788642552.983921 punch from 0x049f97 to 0x04be88 (0x1ef2 bytes) 000424[1] 1788642553.008754 punch from 0x13b3cd to 0x1428a4 (0x74d8 bytes) 000426[0] 1788642553.064004 read 0x17cd01 thru 0x17ee02 (0x2102 bytes) 000428[1] 1788642553.094119 punch from 0x4b0ca8 to 0x4b15f1 (0x94a bytes) 000429[1] 1788642553.111883 trunc from 0x86c50e to 0x0047cb (0x867d42 bytes) 000431[0] 1788642553.151873 mapread 0x00231e thru 0x0033f9 (0x10dc bytes) 000432[0] 1788642553.201561 mapwrite 0x6328cb thru 0x63393a (0x1070 bytes) 000434[1] 1788642553.469500 mapwrite 0x91a857 thru 0x927bf2 (0xd39c bytes) 000435[0] 1788642553.670951 trunc from 0x927bf3 to 0x33358c (0x5f4666 bytes) 000437[0] 1788642553.718988 mapwrite 0x6565c5 thru 0x65990c (0x3348 bytes) 000438[0] 1788642553.798014 write 0x32dd40 thru 0x3399a4 (0xbc65 bytes) 000440[1] 1788642553.849468 punch from 0x290d84 to 0x29299f (0x1c1c bytes) 000441[0] 1788642553.889773 punch from 0x30f835 to 0x31b1d5 (0xb9a1 bytes) 000442[0] 1788642553.934434 mapread 0x3ea265 thru 0x3ebbf5 (0x1991 bytes) 000445[0] 1788642553.962841 mapread 0x011787 thru 0x019b85 (0x83ff bytes) 000447[0] 1788642554.004490 read 0x4d0b76 thru 0x4d597a (0x4e05 bytes) 000448[0] 1788642554.015544 write 0x201153 thru 0x208902 (0x77b0 bytes) 000453[1] 1788642554.048328 punch from 0x2d1712 to 0x2d178c (0x7b bytes) 000455[0] 1788642554.124905 read 0x551c5a thru 0x55e341 (0xc6e8 bytes) 000456[0] 1788642554.152008 punch from 0x6397ea to 0x6442f4 (0xab0b bytes) 000459[1] 1788642554.194474 read 0x50294d thru 0x50e6a5 (0xbd59 bytes) 000460[0] 1788642554.212882 punch from 0x032f23 to 0x0386d3 (0x57b1 bytes) 000462[1] 1788642554.266139 punch from 0x1e5f33 to 0x1eb106 (0x51d4 bytes) 000463[1] 1788642554.301804 read 0x264dcd thru 0x271372 (0xc5a6 bytes) 000464[0] 1788642554.322443 mapwrite 0x3a633f thru 0x3aa25b (0x3f1d bytes) 000465[1] 1788642554.454126 punch from 0x0b46b9 to 0x0bc256 (0x7b9e bytes) 000469[0] 1788642554.506182 mapwrite 0x649d74 thru 0x651b2b (0x7db8 bytes) 000471[1] 1788642554.686806 read 0x5d3c70 thru 0x5e3561 (0xf8f2 bytes) 000473[1] 1788642554.704972 trunc from 0x65990d to 0x9b868b (0x35ed7f bytes) 000475[0] 1788642554.782084 write 0x720798 thru 0x72e476 (0xdcdf bytes) 000476[0] 1788642554.819075 trunc from 0x9b868c to 0x8fdd82 (0xba909 bytes) 000477[0] 1788642554.847198 mapread 0x410cc4 thru 0x4153aa (0x46e7 bytes) 000478[1] 1788642554.900041 mapwrite 0x4e6c47 thru 0x4f1c5b (0xb015 bytes) 000481[1] 1788642555.056279 trunc from 0x8fdd83 to 0x704a72 (0x1f9310 bytes) 000484[1] 1788642555.139132 mapread 0x2ca3b5 thru 0x2cd349 (0x2f95 bytes) 000486[1] 1788642555.202653 mapread 0x39db0a thru 0x39f887 (0x1d7e bytes) 000489[0] 1788642555.344152 write 0x228735 thru 0x23005b (0x7927 bytes) 000490[1] 1788642555.430812 read 0x0232af thru 0x0253a5 (0x20f7 bytes) 000491[0] 1788642555.496343 trunc from 0x704a73 to 0x9d609f (0x2d162d bytes) 000493[1] 1788642555.612034 trunc from 0x9d60a0 to 0x4c5014 (0x51108b bytes) 000494[1] 1788642555.651535 read 0x0443ed thru 0x04988b (0x549f bytes) 000495[0] 1788642555.658927 mapread 0x3686ca thru 0x373788 (0xb0bf bytes) 000498[0] 1788642555.738055 mapread 0x4398b6 thru 0x444c79 (0xb3c4 bytes) 000500[0] 1788642555.781028 trunc from 0x4c5015 to 0x5d3e43 (0x10ee2f bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 2555 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1788642556.536575 write 0x85e000 thru 0x8611d1 (0x31d2 bytes) 000003[0] 1788642556.582500 read 0x350000 thru 0x355ad8 (0x5ad9 bytes) 000006[1] 1788642556.593434 mapread 0x48a000 thru 0x494f53 (0xaf54 bytes) 000007[0] 1788642556.634453 mapwrite 0x5e5000 thru 0x5ee39f (0x93a0 bytes) 000009[1] 1788642556.728936 write 0x8b1000 thru 0x8b3c65 (0x2c66 bytes) 000010[0] 1788642556.740682 mapwrite 0x4c0000 thru 0x4cc226 (0xc227 bytes) 000012[1] 1788642556.894818 mapread 0x276000 thru 0x285b13 (0xfb14 bytes) 000013[1] 1788642556.953097 mapread 0x2ce000 thru 0x2d7c04 (0x9c05 bytes) 000015[0] 1788642556.973847 mapwrite 0x34a000 thru 0x34a7fc (0x7fd bytes) 000016[0] 1788642557.093710 read 0x11e000 thru 0x121632 (0x3633 bytes) 000017[1] 1788642557.101094 mapwrite 0x23f000 thru 0x24aaf2 (0xbaf3 bytes) 000018[0] 1788642557.220654 mapwrite 0x339000 thru 0x33fab2 (0x6ab3 bytes) 000019[0] 1788642557.310725 mapwrite 0x4ea000 thru 0x4ee446 (0x4447 bytes) 000020[1] 1788642557.395670 read 0x6a1000 thru 0x6aa66b (0x966c bytes) 000021[1] 1788642557.418719 mapread 0x1c1000 thru 0x1c8bd1 (0x7bd2 bytes) truncating to largest ever: 0x8153f4 000022[1] 1788642557.461582 trunc from 0x8b3c66 to 0x8153f3 (0x9e872 bytes) 000023[1] 1788642557.521849 mapread 0x1bb000 thru 0x1bbf1c (0xf1d bytes) 000024[1] 1788642557.574837 punch from 0x26939b to 0x275480 (0xc0e6 bytes) 000025[1] 1788642557.597177 punch from 0x0692c9 to 0x077044 (0xdd7c bytes) 000026[0] 1788642557.609527 read 0x3b9000 thru 0x3c5822 (0xc823 bytes) 000027[0] 1788642557.632409 trunc from 0x8153f4 to 0x5b7a5f (0x25d994 bytes) 000028[1] 1788642557.677201 trunc from 0x5b7a60 to 0x21b1cb (0x39c894 bytes) 000029[0] 1788642557.723081 punch from 0x17330c to 0x17b642 (0x8337 bytes) 000031[0] 1788642557.743257 mapwrite 0x77d000 thru 0x7821a2 (0x51a3 bytes) 000032[0] 1788642557.844348 mapwrite 0x048000 thru 0x055f09 (0xdf0a bytes) 000034[0] 1788642558.001518 mapread 0x586000 thru 0x5864d1 (0x4d2 bytes) 000036[0] 1788642558.042466 mapwrite 0x531000 thru 0x53d5e0 (0xc5e1 bytes) 000037[1] 1788642558.097979 write 0x183000 thru 0x187086 (0x4087 bytes) 000039[0] 1788642558.126077 mapread 0x241000 thru 0x25049e (0xf49f bytes) 000042[1] 1788642558.204005 punch from 0x24b0c5 to 0x24f6dd (0x4619 bytes) 000043[0] 1788642558.268335 read 0x228000 thru 0x228469 (0x46a bytes) 000044[0] 1788642558.306885 read 0x0c4000 thru 0x0c9001 (0x5002 bytes) 000045[0] 1788642558.315364 write 0x543000 thru 0x5443ad (0x13ae bytes) 000047[1] 1788642558.353946 read 0x75d000 thru 0x765190 (0x8191 bytes) 000048[1] 1788642558.386772 trunc from 0x7821a3 to 0x2d23d3 (0x4afdcf bytes) 000050[0] 1788642558.435862 mapwrite 0x53e000 thru 0x548251 (0xa252 bytes) 000051[1] 1788642558.584482 trunc from 0x548252 to 0x3166d8 (0x231b79 bytes) 000052[0] 1788642558.624067 write 0x58f000 thru 0x5920ea (0x30eb bytes) 000054[0] 1788642558.662596 read 0x196000 thru 0x19ef74 (0x8f75 bytes) 000058[1] 1788642558.715115 trunc from 0x5920eb to 0x04c67b (0x545a6f bytes) 000060[0] 1788642558.757698 mapwrite 0x638000 thru 0x63b4fd (0x34fe bytes) 000063[1] 1788642558.887071 punch from 0x2ac00f to 0x2b9a22 (0xda14 bytes) 000064[1] 1788642558.915913 write 0x44f000 thru 0x458ad2 (0x9ad3 bytes) 000065[0] 1788642558.925259 mapread 0x59f000 thru 0x5a4175 (0x5176 bytes) 000066[0] 1788642558.966857 mapread 0x363000 thru 0x371d12 (0xed13 bytes) 000068[1] 1788642559.000896 mapread 0x55a000 thru 0x568dc3 (0xedc4 bytes) 000069[1] 1788642559.026669 read 0x4b9000 thru 0x4ba1e2 (0x11e3 bytes) 000070[0] 1788642559.030998 trunc from 0x63b4fe to 0x75e7d4 (0x1232d7 bytes) 000071[1] 1788642559.061672 punch from 0x4a75c8 to 0x4a9dd9 (0x2812 bytes) 000075[1] 1788642559.153604 write 0x7de000 thru 0x7e7e43 (0x9e44 bytes) 000076[0] 1788642559.154911 mapread 0x020000 thru 0x024bcc (0x4bcd bytes) 000077[1] 1788642559.247147 read 0x501000 thru 0x502bc0 (0x1bc1 bytes) 000078[0] 1788642559.255492 mapread 0x345000 thru 0x352a7f (0xda80 bytes) 000079[0] 1788642559.291723 read 0x66b000 thru 0x675712 (0xa713 bytes) 000080[1] 1788642559.304792 trunc from 0x7e7e44 to 0x68f42e (0x158a15 bytes) 000083[1] 1788642559.394902 punch from 0x5ff8d9 to 0x606ad3 (0x71fb bytes) 000084[0] 1788642559.417157 mapread 0x55b000 thru 0x561505 (0x6506 bytes) 000085[0] 1788642559.459359 read 0x084000 thru 0x093b0e (0xfb0f bytes) 000086[1] 1788642559.468132 write 0x1fe000 thru 0x204492 (0x6493 bytes) 000090[1] 1788642559.535941 mapread 0x189000 thru 0x194e76 (0xbe77 bytes) 000092[1] 1788642559.570320 trunc from 0x68f42f to 0x6184b7 (0x76f77 bytes) 000093[1] 1788642559.604992 trunc from 0x6184b8 to 0x149e19 (0x4ce69e bytes) 000094[1] 1788642559.639941 mapwrite 0x3c8000 thru 0x3d0ab2 (0x8ab3 bytes) 000096[0] 1788642559.722272 mapwrite 0x453000 thru 0x45fded (0xcdee bytes) 000097[1] 1788642559.877604 write 0x73d000 thru 0x74691a (0x991b bytes) 000098[1] 1788642559.891373 mapread 0x5de000 thru 0x5ebf40 (0xdf41 bytes) 000102[1] 1788642559.955657 read 0x4d2000 thru 0x4dad29 (0x8d2a bytes) 000103[1] 1788642559.981194 mapwrite 0x098000 thru 0x0a4270 (0xc271 bytes) 000104[1] 1788642560.074432 read 0x0cc000 thru 0x0dbaac (0xfaad bytes) 000105[1] 1788642560.091091 mapwrite 0x98a000 thru 0x9950ed (0xb0ee bytes) 000106[0] 1788642560.245124 read 0x313000 thru 0x31a90a (0x790b bytes) 000108[0] 1788642560.266887 trunc from 0x9950ee to 0x450fd3 (0x54411a bytes) 000111[1] 1788642560.322469 write 0x536000 thru 0x5458d5 (0xf8d6 bytes) 000115[0] 1788642560.381024 write 0x9bb000 thru 0x9bf122 (0x4123 bytes) 000117[0] 1788642560.446582 mapwrite 0x875000 thru 0x884e5e (0xfe5f bytes) 000119[0] 1788642560.548983 read 0x390000 thru 0x39d3ec (0xd3ed bytes) 000120[0] 1788642560.562161 read 0x245000 thru 0x249ba6 (0x4ba7 bytes) 000121[1] 1788642560.589670 read 0x8e2000 thru 0x8f1055 (0xf056 bytes) 000123[1] 1788642560.615882 mapwrite 0x63c000 thru 0x63f189 (0x318a bytes) 000124[0] 1788642560.731374 mapwrite 0x04a000 thru 0x04b57e (0x157f bytes) 000127[0] 1788642560.839873 write 0x917000 thru 0x91ca69 (0x5a6a bytes) 000128[0] 1788642560.845296 mapread 0x56e000 thru 0x56e555 (0x556 bytes) 000129[1] 1788642560.872063 mapread 0x7a3000 thru 0x7ab06d (0x806e bytes) 000130[0] 1788642560.936247 write 0x0ce000 thru 0x0d13c6 (0x33c7 bytes) 000136[0] 1788642561.078527 mapread 0x26f000 thru 0x27552a (0x652b bytes) truncating to largest ever: 0x8cf7f6 000140[1] 1788642561.158963 trunc from 0x9bf123 to 0x8cf7f5 (0xef92d bytes) 000143[0] 1788642561.207856 read 0x898000 thru 0x89d8fe (0x58ff bytes) 000144[1] 1788642561.236099 punch from 0x668f83 to 0x6744f5 (0xb573 bytes) 000145[1] 1788642561.250733 mapread 0x7df000 thru 0x7ee879 (0xf87a bytes) 000146[0] 1788642561.274942 write 0x819000 thru 0x81958c (0x58d bytes) 000147[1] 1788642561.294838 punch from 0x02ba32 to 0x03a813 (0xede2 bytes) 000148[1] 1788642561.328105 write 0x3d5000 thru 0x3dd484 (0x8485 bytes) 000149[1] 1788642561.335162 mapread 0x4d4000 thru 0x4e3ae4 (0xfae5 bytes) 000152[1] 1788642561.388058 mapwrite 0x5e8000 thru 0x5f518f (0xd190 bytes) truncating to largest ever: 0x9d74aa 000153[1] 1788642561.452637 trunc from 0x8cf7f6 to 0x9d74a9 (0x107cb4 bytes) 000155[0] 1788642561.501628 write 0x5e4000 thru 0x5e4094 (0x95 bytes) 000156[1] 1788642561.516907 write 0x3df000 thru 0x3edcec (0xeced bytes) 000157[1] 1788642561.556187 punch from 0x66d30f to 0x66d5cf (0x2c1 bytes) 000159[0] 1788642561.596847 mapread 0x68f000 thru 0x69a6fb (0xb6fc bytes) 000161[0] 1788642561.707182 mapwrite 0x90a000 thru 0x90b5d7 (0x15d8 bytes) 000163[0] 1788642561.787409 mapwrite 0x4d8000 thru 0x4e2fcd (0xafce bytes) 000166[1] 1788642561.879802 mapread 0x985000 thru 0x992762 (0xd763 bytes) 000167[0] 1788642561.933274 punch from 0x9499e1 to 0x94f68b (0x5cab bytes) 000168[0] 1788642561.958243 read 0x991000 thru 0x9987e7 (0x77e8 bytes) 000169[0] 1788642561.965992 punch from 0x8fb29d to 0x902e4a (0x7bae bytes) 000170[0] 1788642561.987399 trunc from 0x9d74aa to 0x8a2b4e (0x13495b bytes) 000171[1] 1788642562.012850 read 0x11b000 thru 0x120f35 (0x5f36 bytes) 000172[0] 1788642562.047170 read 0x363000 thru 0x364840 (0x1841 bytes) 000175[1] 1788642562.058198 punch from 0x42e7a6 to 0x439851 (0xb0ac bytes) 000177[1] 1788642562.088055 write 0x0fe000 thru 0x102d47 (0x4d48 bytes) 000179[1] 1788642562.137409 mapread 0x492000 thru 0x4989fb (0x69fc bytes) 000181[0] 1788642562.234088 mapread 0x3f2000 thru 0x3f4c3b (0x2c3c bytes) 000182[0] 1788642562.281620 mapread 0x04c000 thru 0x04f189 (0x318a bytes) 000183[0] 1788642562.315082 punch from 0x688a72 to 0x68cb25 (0x40b4 bytes) 000184[1] 1788642562.352170 read 0x7b7000 thru 0x7c6f58 (0xff59 bytes) 000185[1] 1788642562.377877 trunc from 0x8a2b4f to 0x99828b (0xf573d bytes) 000187[0] 1788642562.442804 trunc from 0x99828c to 0x7266b3 (0x271bd8 bytes) 000188[1] 1788642562.482800 punch from 0x4bb399 to 0x4c2517 (0x717f bytes) 000193[1] 1788642562.552737 mapread 0x2b8000 thru 0x2bf701 (0x7702 bytes) 000194[1] 1788642562.591176 read 0x207000 thru 0x210f14 (0x9f15 bytes) 000195[0] 1788642562.599939 write 0x00c000 thru 0x015224 (0x9225 bytes) 000196[0] 1788642562.631796 mapwrite 0x4a7000 thru 0x4b27d5 (0xb7d6 bytes) 000198[1] 1788642562.711818 read 0x594000 thru 0x5a30cd (0xf0ce bytes) 000199[1] 1788642562.730784 punch from 0x364767 to 0x3692e6 (0x4b80 bytes) 000200[1] 1788642562.747955 punch from 0x4d88af to 0x4dea5e (0x61b0 bytes) 000201[1] 1788642562.753716 mapwrite 0x24e000 thru 0x24f979 (0x197a bytes) 000202[0] 1788642562.796415 mapread 0x12c000 thru 0x138d7f (0xcd80 bytes) 000203[0] 1788642562.845313 read 0x5e9000 thru 0x5e9cf9 (0xcfa bytes) 000204[1] 1788642562.850571 mapread 0x0b6000 thru 0x0c544c (0xf44d bytes) 000206[0] 1788642562.881843 punch from 0x37e735 to 0x383b01 (0x53cd bytes) 000207[0] 1788642562.903717 mapwrite 0x0eb000 thru 0x0f0121 (0x5122 bytes) 000208[0] 1788642563.011878 mapread 0x247000 thru 0x247675 (0x676 bytes) 000209[1] 1788642563.044344 mapread 0x4a3000 thru 0x4b1de9 (0xedea bytes) 000210[1] 1788642563.120025 write 0x304000 thru 0x312491 (0xe492 bytes) 000211[0] 1788642563.137615 write 0x3ea000 thru 0x3f228f (0x8290 bytes) 000213[1] 1788642563.160337 write 0x8e9000 thru 0x8eba3f (0x2a40 bytes) 000214[0] 1788642563.196808 mapwrite 0x105000 thru 0x10ed0e (0x9d0f bytes) 000216[1] 1788642563.430971 trunc from 0x8eba40 to 0x241d84 (0x6a9cbb bytes) 000217[1] 1788642563.528083 mapwrite 0x9a2000 thru 0x9a8d98 (0x6d99 bytes) 000220[0] 1788642563.674662 trunc from 0x9a8d99 to 0x6d539f (0x2d39f9 bytes) 000221[1] 1788642563.742145 write 0x4ee000 thru 0x4fcf41 (0xef42 bytes) 000222[0] 1788642563.787654 mapread 0x5d2000 thru 0x5da9a2 (0x89a3 bytes) 000223[0] 1788642563.840775 trunc from 0x6d53a0 to 0x34bcda (0x3896c5 bytes) 000224[1] 1788642563.869784 write 0x826000 thru 0x82c880 (0x6881 bytes) 000226[1] 1788642563.916084 write 0x3b1000 thru 0x3bf1ad (0xe1ae bytes) 000227[1] 1788642563.927844 trunc from 0x82c881 to 0x7ac9f6 (0x7fe8a bytes) 000228[1] 1788642563.971945 punch from 0x4541a8 to 0x46235e (0xe1b7 bytes) 000232[0] 1788642564.054410 read 0x18a000 thru 0x1963f6 (0xc3f7 bytes) 000233[0] 1788642564.087156 mapwrite 0x77e000 thru 0x77f248 (0x1249 bytes) 000234[0] 1788642564.174921 write 0x930000 thru 0x934f46 (0x4f47 bytes) 000235[0] 1788642564.177011 mapwrite 0x3a5000 thru 0x3b3f35 (0xef36 bytes) 000236[1] 1788642564.258978 read 0x11c000 thru 0x11e32f (0x2330 bytes) 000237[1] 1788642564.279591 read 0x876000 thru 0x8775c3 (0x15c4 bytes) 000240[1] 1788642564.285968 read 0x6e3000 thru 0x6f14aa (0xe4ab bytes) 000241[1] 1788642564.307584 mapwrite 0x80c000 thru 0x81b323 (0xf324 bytes) 000242[1] 1788642564.423552 mapread 0x099000 thru 0x09fbdf (0x6be0 bytes) 000243[1] 1788642564.456700 punch from 0x2a62a8 to 0x2b507b (0xedd4 bytes) 000245[1] 1788642564.479070 mapread 0x500000 thru 0x50edfe (0xedff bytes) 000249[1] 1788642564.537756 mapread 0x450000 thru 0x456c0e (0x6c0f bytes) 000250[0] 1788642564.552148 trunc from 0x934f47 to 0x355867 (0x5df6df bytes) 000251[1] 1788642564.585846 punch from 0x0c8c35 to 0x0cd266 (0x4632 bytes) 000253[1] 1788642564.629039 punch from 0x0e02d2 to 0x0e6d8d (0x6abc bytes) 000254[0] 1788642564.648993 read 0x071000 thru 0x07a56e (0x956f bytes) 000256[0] 1788642564.739687 punch from 0x2e36d9 to 0x2e9c4d (0x6575 bytes) 000257[0] 1788642564.760955 mapread 0x2a2000 thru 0x2acb47 (0xab48 bytes) 000258[1] 1788642564.829926 mapread 0x00e000 thru 0x012717 (0x4718 bytes) 000260[1] 1788642564.898173 punch from 0x1a0fc7 to 0x1a9cf8 (0x8d32 bytes) 000265[0] 1788642564.953977 punch from 0x24ebe3 to 0x2566a4 (0x7ac2 bytes) 000267[1] 1788642564.990900 punch from 0x335a75 to 0x33cf62 (0x74ee bytes) 000268[1] 1788642565.009312 mapwrite 0x479000 thru 0x47cafc (0x3afd bytes) 000269[1] 1788642565.129317 write 0x8e4000 thru 0x8e85aa (0x45ab bytes) 000271[0] 1788642565.135731 mapread 0x055000 thru 0x062fda (0xdfdb bytes) 000272[1] 1788642565.298586 write 0x51d000 thru 0x51ee15 (0x1e16 bytes) 000274[0] 1788642565.357163 mapwrite 0x542000 thru 0x542261 (0x262 bytes) 000275[1] 1788642565.522067 mapread 0x386000 thru 0x38d45b (0x745c bytes) 000276[1] 1788642565.573783 punch from 0x3988f9 to 0x3a35ca (0xacd2 bytes) 000277[1] 1788642565.588272 write 0x2f6000 thru 0x3047e5 (0xe7e6 bytes) 000278[1] 1788642565.604043 read 0x09f000 thru 0x0a4235 (0x5236 bytes) 000279[1] 1788642565.611530 punch from 0x515a72 to 0x52212e (0xc6bd bytes) 000281[1] 1788642565.644703 mapwrite 0x86f000 thru 0x876ffd (0x7ffe bytes) 000282[1] 1788642565.722089 read 0x09e000 thru 0x0a5a15 (0x7a16 bytes) 000283[1] 1788642565.731031 mapread 0x136000 thru 0x139ade (0x3adf bytes) 000284[1] 1788642565.754948 write 0x0f4000 thru 0x0fabb7 (0x6bb8 bytes) 000285[0] 1788642565.787282 write 0x13d000 thru 0x147023 (0xa024 bytes) 000288[0] 1788642565.837954 mapwrite 0x8a2000 thru 0x8b19d4 (0xf9d5 bytes) 000289[1] 1788642565.913045 read 0x3e0000 thru 0x3ee970 (0xe971 bytes) 000292[1] 1788642565.940774 write 0x533000 thru 0x53ab21 (0x7b22 bytes) 000293[0] 1788642565.954760 punch from 0x75a18a to 0x75f158 (0x4fcf bytes) 000298[1] 1788642566.037970 punch from 0x8c1f52 to 0x8c4318 (0x23c7 bytes) 000299[0] 1788642566.068130 punch from 0x00caa8 to 0x017d96 (0xb2ef bytes) 000300[1] 1788642566.086322 mapread 0x553000 thru 0x557b66 (0x4b67 bytes) 000303[1] 1788642566.127737 trunc from 0x8e85ab to 0x441587 (0x4a7023 bytes) 000304[0] 1788642566.172044 write 0x0e9000 thru 0x0e9a07 (0xa08 bytes) 000307[0] 1788642566.264031 mapwrite 0x17f000 thru 0x185771 (0x6772 bytes) 000308[0] 1788642566.355558 mapwrite 0x84b000 thru 0x84f1a1 (0x41a2 bytes) 000312[1] 1788642566.470061 read 0x5a9000 thru 0x5ad34e (0x434f bytes) 000313[0] 1788642566.490340 mapwrite 0x40b000 thru 0x41684a (0xb84b bytes) 000316[0] 1788642566.657926 mapread 0x279000 thru 0x27cba7 (0x3ba8 bytes) 000321[0] 1788642566.691608 punch from 0x37ee89 to 0x37f0b7 (0x22f bytes) 000324[0] 1788642566.728859 mapread 0x81e000 thru 0x81e78e (0x78f bytes) 000326[0] 1788642566.767738 mapwrite 0x479000 thru 0x4807da (0x77db bytes) 000328[0] 1788642566.908947 mapwrite 0x0af000 thru 0x0bb80e (0xc80f bytes) 000333[1] 1788642567.052000 trunc from 0x84f1a2 to 0x586697 (0x2c8b0a bytes) 000335[1] 1788642567.137679 read 0x4a7000 thru 0x4b0cea (0x9ceb bytes) 000337[0] 1788642567.160710 write 0x7bb000 thru 0x7bb86f (0x870 bytes) 000339[1] 1788642567.201901 trunc from 0x7bb870 to 0x022d9f (0x798ad0 bytes) 000340[1] 1788642567.248632 write 0x968000 thru 0x970094 (0x8095 bytes) 000342[0] 1788642567.310943 trunc from 0x970095 to 0x45ce0a (0x51328a bytes) 000343[1] 1788642567.355098 write 0x443000 thru 0x44aff1 (0x7ff2 bytes) 000344[1] 1788642567.392460 read 0x0c1000 thru 0x0c919b (0x819c bytes) 000347[1] 1788642567.404701 mapwrite 0x16a000 thru 0x16cabb (0x2abc bytes) 000348[1] 1788642567.496166 mapwrite 0x5d2000 thru 0x5e16da (0xf6db bytes) 000349[0] 1788642567.657873 mapread 0x57c000 thru 0x57fc3c (0x3c3d bytes) 000350[0] 1788642567.705848 read 0x2bf000 thru 0x2cc530 (0xd531 bytes) 000351[0] 1788642567.715295 trunc from 0x5e16db to 0x835df5 (0x25471b bytes) 000352[1] 1788642567.780685 mapread 0x808000 thru 0x814424 (0xc425 bytes) 000353[1] 1788642567.844069 write 0x7a1000 thru 0x7aeb80 (0xdb81 bytes) 000354[0] 1788642567.882020 mapread 0x4b1000 thru 0x4bfbe6 (0xebe7 bytes) 000355[1] 1788642567.937820 trunc from 0x835df6 to 0x0abfd2 (0x789e23 bytes) 000356[0] 1788642567.982821 write 0x683000 thru 0x68c12c (0x912d bytes) 000357[0] 1788642568.006551 punch from 0x21fbb4 to 0x221414 (0x1861 bytes) 000358[0] 1788642568.011121 mapread 0x3fa000 thru 0x406298 (0xc299 bytes) 000359[0] 1788642568.025016 write 0x77b000 thru 0x77fa80 (0x4a81 bytes) 000360[0] 1788642568.036638 mapwrite 0x003000 thru 0x0064b1 (0x34b2 bytes) 000361[1] 1788642568.106956 mapread 0x77e000 thru 0x77f4a6 (0x14a7 bytes) 000362[0] 1788642568.131205 punch from 0x5a2e79 to 0x5a91cc (0x6354 bytes) 000365[1] 1788642568.162919 punch from 0x6c8f98 to 0x6cf243 (0x62ac bytes) 000366[0] 1788642568.190277 read 0x47a000 thru 0x48018f (0x6190 bytes) 000367[0] 1788642568.209752 read 0x3a5000 thru 0x3a8632 (0x3633 bytes) 000369[1] 1788642568.219975 mapread 0x29e000 thru 0x2a3721 (0x5722 bytes) 000370[0] 1788642568.242478 mapwrite 0x1a8000 thru 0x1b6d22 (0xed23 bytes) 000371[1] 1788642568.332330 punch from 0x2a2625 to 0x2ad2c0 (0xac9c bytes) 000372[1] 1788642568.348945 write 0x35f000 thru 0x365463 (0x6464 bytes) 000376[1] 1788642568.390682 read 0x6d6000 thru 0x6e3446 (0xd447 bytes) 000377[1] 1788642568.400028 mapwrite 0x6d7000 thru 0x6e3902 (0xc903 bytes) 000378[0] 1788642568.467130 read 0x66b000 thru 0x66ea3f (0x3a40 bytes) 000379[1] 1788642568.493790 trunc from 0x77fa81 to 0x64ba28 (0x134058 bytes) 000381[1] 1788642568.553319 read 0x16f000 thru 0x1778d6 (0x88d7 bytes) 000382[0] 1788642568.567510 read 0x279000 thru 0x28873b (0xf73c bytes) 000383[1] 1788642568.596831 punch from 0x493c67 to 0x49be35 (0x81cf bytes) 000384[1] 1788642568.637544 punch from 0x1e7570 to 0x1f1a53 (0xa4e4 bytes) 000385[0] 1788642568.659967 read 0x1b2000 thru 0x1b43cf (0x23d0 bytes) 000386[0] 1788642568.688140 trunc from 0x64ba29 to 0x6de548 (0x92b20 bytes) 000387[1] 1788642568.739970 mapread 0x26b000 thru 0x26bc7c (0xc7d bytes) 000388[0] 1788642568.802139 write 0x91a000 thru 0x928539 (0xe53a bytes) 000389[0] 1788642568.837488 write 0x1bf000 thru 0x1ce4e0 (0xf4e1 bytes) 000390[1] 1788642568.861411 mapread 0x02f000 thru 0x03aa3a (0xba3b bytes) 000393[1] 1788642568.911947 trunc from 0x92853a to 0x4af63a (0x478eff bytes) 000398[1] 1788642568.970245 mapwrite 0x4ac000 thru 0x4bb8e0 (0xf8e1 bytes) 000400[0] 1788642569.123771 mapwrite 0x16e000 thru 0x173067 (0x5068 bytes) 000402[0] 1788642569.244679 trunc from 0x4bb8e1 to 0x1b5895 (0x30604b bytes) 000403[1] 1788642569.293471 mapwrite 0x09e000 thru 0x0a5243 (0x7244 bytes) 000405[1] 1788642569.428321 punch from 0x0898a4 to 0x08e922 (0x507f bytes) 000406[1] 1788642569.469985 write 0x218000 thru 0x21946d (0x146e bytes) 000407[0] 1788642569.497846 mapwrite 0x617000 thru 0x61cb11 (0x5b12 bytes) 000408[0] 1788642569.671057 mapwrite 0x5fd000 thru 0x605443 (0x8444 bytes) 000409[1] 1788642569.767827 mapread 0x4ee000 thru 0x4f3fec (0x5fed bytes) 000412[0] 1788642569.821444 mapwrite 0x9c0000 thru 0x9cbc0c (0xbc0d bytes) 000413[0] 1788642569.938582 write 0x3a9000 thru 0x3a9d39 (0xd3a bytes) 000414[1] 1788642569.949047 punch from 0x831ab2 to 0x832bd2 (0x1121 bytes) 000417[1] 1788642569.988975 write 0x0c9000 thru 0x0ce059 (0x505a bytes) 000419[1] 1788642570.021716 trunc from 0x9cbc0d to 0x3cc8f8 (0x5ff314 bytes) 000420[0] 1788642570.056925 mapwrite 0x67c000 thru 0x67e40c (0x240d bytes) 000421[0] 1788642570.143852 punch from 0x086c4a to 0x08a0cc (0x3483 bytes) 000422[0] 1788642570.163176 read 0x4cb000 thru 0x4d0485 (0x5486 bytes) 000423[0] 1788642570.183685 mapread 0x4b0000 thru 0x4b2147 (0x2148 bytes) 000424[1] 1788642570.200498 write 0x140000 thru 0x141bb3 (0x1bb4 bytes) 000426[1] 1788642570.231631 read 0x62d000 thru 0x633904 (0x6905 bytes) 000427[1] 1788642570.240317 read 0x5e4000 thru 0x5ea5f6 (0x65f7 bytes) 000429[0] 1788642570.247923 mapread 0x629000 thru 0x62ac60 (0x1c61 bytes) 000430[0] 1788642570.279429 punch from 0x1a1580 to 0x1a9ed8 (0x8959 bytes) 000431[0] 1788642570.294072 trunc from 0x67e40d to 0x0d5b13 (0x5a88f9 bytes) 000433[0] 1788642570.337015 trunc from 0x0d5b14 to 0x6cd2df (0x5f77cc bytes) 000434[1] 1788642570.356763 read 0x17d000 thru 0x183397 (0x6398 bytes) 000435[0] 1788642570.481100 trunc from 0x6cd2e0 to 0x7bce37 (0xefb58 bytes) 000436[1] 1788642570.596153 mapwrite 0x623000 thru 0x62bb53 (0x8b54 bytes) 000437[0] 1788642570.838979 mapread 0x3bd000 thru 0x3c5fb2 (0x8fb3 bytes) 000440[1] 1788642570.888126 read 0x7b7000 thru 0x7bc918 (0x5919 bytes) 000445[0] 1788642570.900597 mapwrite 0x803000 thru 0x805fdb (0x2fdc bytes) 000446[1] 1788642571.051131 write 0x5ca000 thru 0x5cdaa7 (0x3aa8 bytes) 000447[0] 1788642571.069829 mapwrite 0x257000 thru 0x25724d (0x24e bytes) 000449[1] 1788642571.154938 mapread 0x620000 thru 0x623a6e (0x3a6f bytes) 000451[0] 1788642571.203406 trunc from 0x805fdc to 0x784e0b (0x811d0 bytes) 000452[1] 1788642571.243541 mapwrite 0x5ca000 thru 0x5d19a1 (0x79a2 bytes) 000453[0] 1788642571.344932 mapwrite 0x8b1000 thru 0x8b335a (0x235b bytes) 000455[1] 1788642571.530123 read 0x767000 thru 0x767713 (0x714 bytes) 000456[0] 1788642571.552875 mapread 0x510000 thru 0x5139e0 (0x39e1 bytes) 000457[0] 1788642571.578963 mapwrite 0x237000 thru 0x23e267 (0x7268 bytes) 000458[1] 1788642571.712988 write 0x528000 thru 0x535014 (0xd015 bytes) 000460[1] 1788642571.779496 trunc from 0x8b335b to 0x0c73b4 (0x7ebfa6 bytes) 000461[0] 1788642571.814995 trunc from 0x0c73b5 to 0x91fbf0 (0x85883c bytes) 000462[0] 1788642571.841454 punch from 0x805397 to 0x8062b2 (0xf1c bytes) 000464[0] 1788642571.873165 read 0x7ed000 thru 0x7f081e (0x381f bytes) 000465[0] 1788642571.879993 punch from 0x21b527 to 0x21fd16 (0x47f0 bytes) 000466[0] 1788642571.884286 write 0x948000 thru 0x9570e2 (0xf0e3 bytes) 000467[1] 1788642571.896818 trunc from 0x9570e3 to 0x7d6237 (0x180eab bytes) 000469[0] 1788642571.962059 write 0x8a7000 thru 0x8b104a (0xa04b bytes) 000470[1] 1788642571.996977 mapread 0x769000 thru 0x7777c3 (0xe7c4 bytes) 000472[1] 1788642572.052838 mapread 0x103000 thru 0x10c594 (0x9595 bytes) 000473[0] 1788642572.069873 read 0x848000 thru 0x851b15 (0x9b16 bytes) 000475[0] 1788642572.078661 write 0x5e8000 thru 0x5f22f6 (0xa2f7 bytes) 000476[0] 1788642572.118327 trunc from 0x8b104b to 0x2a4342 (0x60cd08 bytes) 000477[0] 1788642572.147546 read 0x145000 thru 0x14a9ef (0x59f0 bytes) 000479[0] 1788642572.196983 mapread 0x176000 thru 0x17a30e (0x430f bytes) 000480[0] 1788642572.217200 mapread 0x075000 thru 0x07729d (0x229e bytes) 000482[0] 1788642572.265777 write 0x6f7000 thru 0x7026ef (0xb6f0 bytes) 000485[0] 1788642572.328472 trunc from 0x7026f0 to 0x93b844 (0x239155 bytes) 000488[1] 1788642572.372628 punch from 0x37772c to 0x37d2fd (0x5bd2 bytes) 000490[1] 1788642572.419661 read 0x819000 thru 0x81e226 (0x5227 bytes) 000491[1] 1788642572.427869 punch from 0x063b64 to 0x072d3c (0xf1d9 bytes) 000495[0] 1788642572.447842 trunc from 0x93b845 to 0x7af440 (0x18c404 bytes) 000496[0] 1788642572.499725 read 0x3a3000 thru 0x3add6e (0xad6f bytes) 000497[1] 1788642572.520998 write 0x898000 thru 0x89a5e5 (0x25e6 bytes) 000498[1] 1788642572.572882 write 0x4b1000 thru 0x4bfe0f (0xee10 bytes) 000499[1] 1788642572.581768 punch from 0x73e582 to 0x746fb7 (0x8a36 bytes) All operations completed A-OK! PASS 16a (46s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 17:09:38 (1788642578) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.36747 s, 22.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.314958 s, 26.6 MB/s Chance of close/open is 1 in 50 Seed set to 2581 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1788642582.476736 mapwrite 0x977ddc thru 0x97861e (0x843 bytes) 000003[0] 1788642582.633932 mapread 0x847d67 thru 0x849cb5 (0x1f4f bytes) 000007[1] 1788642582.655738 read 0x200d96 thru 0x201663 (0x8ce bytes) truncating to largest ever: 0x9ee2ab 000008[1] 1788642582.678381 trunc from 0x97861f to 0x9ee2aa (0x75c8c bytes) 000011[1] 1788642582.760037 punch from 0x866b59 to 0x868a8e (0x1f36 bytes) 000012[1] 1788642582.765042 write 0x2f6ecc thru 0x2f71f2 (0x327 bytes) 000013[1] 1788642582.800676 punch from 0x1c7d55 to 0x1c7fd1 (0x27d bytes) 000015[0] 1788642582.845767 mapread 0x4c29b4 thru 0x4c2f9b (0x5e8 bytes) 000017[1] 1788642582.903853 mapwrite 0x9d5da2 thru 0x9d5df9 (0x58 bytes) 000018[0] 1788642582.961838 punch from 0x05f820 to 0x060aff (0x12e0 bytes) 000019[0] 1788642582.986709 mapread 0x4b6dc7 thru 0x4b6e21 (0x5b bytes) 000021[0] 1788642583.028566 mapwrite 0x404e14 thru 0x405d65 (0xf52 bytes) 000022[1] 1788642583.209821 punch from 0x9e157c to 0x9e2df5 (0x187a bytes) 000024[1] 1788642583.271770 mapwrite 0x67803b thru 0x67807a (0x40 bytes) 000025[1] 1788642583.366171 mapread 0x490be7 thru 0x490c94 (0xae bytes) 000026[0] 1788642583.372927 punch from 0x1633d8 to 0x164788 (0x13b1 bytes) 000027[0] 1788642583.408712 mapread 0x969967 thru 0x96aaa4 (0x113e bytes) 000028[1] 1788642583.447925 mapwrite 0x5b3fae thru 0x5b47d9 (0x82c bytes) 000030[0] 1788642583.639254 trunc from 0x9ee2ab to 0x7d72b6 (0x216ff4 bytes) 000031[0] 1788642583.696597 read 0x44c846 thru 0x44c871 (0x2c bytes) 000032[1] 1788642583.761683 mapwrite 0x6dfd6f thru 0x6e03b3 (0x645 bytes) 000034[0] 1788642583.884487 punch from 0x5f5a0c to 0x5f6fe6 (0x15db bytes) 000036[1] 1788642583.971260 mapread 0x020654 thru 0x022434 (0x1de1 bytes) 000039[1] 1788642584.027443 mapwrite 0x66baa5 thru 0x66c43b (0x997 bytes) 000040[1] 1788642584.188812 trunc from 0x7d72b7 to 0x1b40ac (0x62320a bytes) 000041[1] 1788642584.255897 punch from 0x139535 to 0x13b1e4 (0x1cb0 bytes) 000043[1] 1788642584.264548 mapread 0x0eb026 thru 0x0eb5f8 (0x5d3 bytes) 000044[0] 1788642584.314606 punch from 0x04da43 to 0x04f84e (0x1e0c bytes) 000045[1] 1788642584.378068 trunc from 0x1b40ad to 0x6d507f (0x520fd3 bytes) 000046[0] 1788642584.466543 read 0x2db535 thru 0x2dc8cf (0x139b bytes) 000047[1] 1788642584.605570 read 0x2d847b thru 0x2d9223 (0xda9 bytes) 000048[0] 1788642584.631647 mapwrite 0x08979b thru 0x089ffc (0x862 bytes) 000049[1] 1788642584.861460 read 0x46c767 thru 0x46e459 (0x1cf3 bytes) 000050[1] 1788642584.908857 mapread 0x53b2dd thru 0x53d284 (0x1fa8 bytes) 000051[0] 1788642584.958413 mapwrite 0x087da9 thru 0x088689 (0x8e1 bytes) 000052[1] 1788642585.067284 trunc from 0x6d5080 to 0x56fed1 (0x1651ae bytes) 000053[1] 1788642585.141937 trunc from 0x56fed2 to 0x0956b3 (0x4da81e bytes) 000055[0] 1788642585.246665 mapread 0x01f1fc thru 0x020ca8 (0x1aad bytes) 000056[0] 1788642585.303937 mapwrite 0x4cd9c1 thru 0x4cee14 (0x1454 bytes) 000058[0] 1788642585.579739 write 0x59247e thru 0x592efe (0xa81 bytes) truncating to largest ever: 0x9f3e28 000059[1] 1788642585.613758 trunc from 0x592eff to 0x9f3e27 (0x460f29 bytes) 000061[0] 1788642585.735600 trunc from 0x9f3e28 to 0x8fc497 (0xf7990 bytes) 000062[0] 1788642585.884746 punch from 0x3566f8 to 0x357e15 (0x171e bytes) 000065[1] 1788642585.896728 mapwrite 0x3895f7 thru 0x38ae45 (0x184f bytes) 000068[1] 1788642586.068848 mapwrite 0x5ea384 thru 0x5eb8a1 (0x151e bytes) 000069[1] 1788642586.288007 read 0x497409 thru 0x497daf (0x9a7 bytes) 000070[0] 1788642586.294164 mapread 0x21a330 thru 0x21baf9 (0x17ca bytes) 000071[0] 1788642586.367761 write 0x0115c0 thru 0x013543 (0x1f84 bytes) 000072[0] 1788642586.432609 trunc from 0x8fc498 to 0x997753 (0x9b2bc bytes) 000076[0] 1788642586.618287 trunc from 0x997754 to 0x1b5432 (0x7e2321 bytes) 000080[1] 1788642586.745817 read 0x0fd4d5 thru 0x0fde2e (0x95a bytes) 000081[1] 1788642586.799112 mapread 0x0502d4 thru 0x050727 (0x454 bytes) 000083[0] 1788642586.835250 mapwrite 0x7cdab9 thru 0x7cf97e (0x1ec6 bytes) 000085[1] 1788642586.931971 trunc from 0x7cf97f to 0x18010c (0x64f872 bytes) 000087[1] 1788642587.022306 write 0x6549ed thru 0x654f01 (0x515 bytes) 000090[1] 1788642587.073744 punch from 0x184d92 to 0x1852f3 (0x562 bytes) 000092[1] 1788642587.148443 write 0x485f76 thru 0x4878bd (0x1948 bytes) 000095[0] 1788642587.179779 write 0x0a0095 thru 0x0a1531 (0x149d bytes) 000096[1] 1788642587.213724 mapread 0x6266e4 thru 0x627f13 (0x1830 bytes) 000097[0] 1788642587.242400 write 0x43fa59 thru 0x4414f5 (0x1a9d bytes) 000099[1] 1788642587.333346 mapread 0x5210a1 thru 0x521aeb (0xa4b bytes) 000100[1] 1788642587.422323 read 0x131190 thru 0x1317e0 (0x651 bytes) 000101[1] 1788642587.463695 trunc from 0x654f02 to 0x48ffc5 (0x1c4f3c bytes) 000102[1] 1788642587.584743 read 0x3c531e thru 0x3c68a4 (0x1587 bytes) 000105[0] 1788642587.640616 read 0x1fff56 thru 0x200de9 (0xe94 bytes) 000106[1] 1788642587.674638 mapwrite 0x2d2b8e thru 0x2d3003 (0x476 bytes) 000107[0] 1788642587.777370 mapread 0x0e4018 thru 0x0e41be (0x1a7 bytes) 000108[0] 1788642587.803164 punch from 0x30d63f to 0x30ec36 (0x15f8 bytes) 000110[1] 1788642587.852912 mapwrite 0x1eef8f thru 0x1efc6f (0xce1 bytes) 000111[0] 1788642587.922395 trunc from 0x48ffc6 to 0x8d61b6 (0x4461f1 bytes) 000112[0] 1788642587.973576 mapwrite 0x6416d3 thru 0x6436a8 (0x1fd6 bytes) 000114[1] 1788642588.061801 read 0x7a4997 thru 0x7a5561 (0xbcb bytes) 000115[0] 1788642588.089743 punch from 0x8947ac to 0x8955bb (0xe10 bytes) 000118[0] 1788642588.134767 write 0x597caf thru 0x5983c3 (0x715 bytes) 000119[1] 1788642588.153583 punch from 0x38e2ef to 0x38f3d7 (0x10e9 bytes) 000120[1] 1788642588.180747 trunc from 0x8d61b7 to 0x9eaf60 (0x114daa bytes) 000121[0] 1788642588.242885 trunc from 0x9eaf61 to 0x9687d4 (0x8278c bytes) 000123[1] 1788642588.303681 punch from 0x3b93ad to 0x3ba126 (0xd7a bytes) 000124[1] 1788642588.333602 punch from 0x88dd0b to 0x88f3a1 (0x1697 bytes) 000125[0] 1788642588.354194 punch from 0x917f10 to 0x91978a (0x187b bytes) 000126[0] 1788642588.372723 punch from 0x6220cc to 0x623678 (0x15ad bytes) 000131[1] 1788642588.521186 read 0x666904 thru 0x6674ed (0xbea bytes) 000132[1] 1788642588.586519 write 0x0af987 thru 0x0b0f86 (0x1600 bytes) 000133[1] 1788642588.629639 write 0x510a63 thru 0x511e20 (0x13be bytes) 000134[0] 1788642588.692919 trunc from 0x9687d5 to 0x3d1b72 (0x596c62 bytes) 000135[0] 1788642588.808033 write 0x6a4d0f thru 0x6a4e1f (0x111 bytes) 000136[0] 1788642588.815302 mapwrite 0x93353c thru 0x9353f2 (0x1eb7 bytes) 000137[1] 1788642588.934289 trunc from 0x9353f3 to 0x072b07 (0x8c28eb bytes) 000144[1] 1788642589.106577 mapwrite 0x9e0b7d thru 0x9e1f72 (0x13f6 bytes) 000146[0] 1788642589.325770 punch from 0x6d31aa to 0x6d5181 (0x1fd8 bytes) 000151[0] 1788642589.422864 write 0x45a620 thru 0x45add9 (0x7ba bytes) 000152[1] 1788642589.442495 mapread 0x0ecd58 thru 0x0ee28a (0x1533 bytes) 000153[0] 1788642589.481799 write 0x94a418 thru 0x94a78a (0x373 bytes) 000157[0] 1788642589.585083 read 0x481282 thru 0x482715 (0x1494 bytes) 000158[1] 1788642589.590557 trunc from 0x9e1f73 to 0x959c46 (0x8832c bytes) 000159[1] 1788642589.642076 write 0x0dcc67 thru 0x0ddf4c (0x12e6 bytes) 000162[0] 1788642589.719198 punch from 0x52a468 to 0x52b702 (0x129b bytes) 000163[1] 1788642589.752903 trunc from 0x959c47 to 0x9f2e4d (0x99207 bytes) 000164[1] 1788642589.804429 mapwrite 0x44341b thru 0x444025 (0xc0b bytes) 000165[0] 1788642589.862183 write 0x1a363f thru 0x1a5306 (0x1cc8 bytes) 000166[1] 1788642589.913930 write 0x54918b thru 0x54a9f6 (0x186c bytes) 000169[0] 1788642589.987435 trunc from 0x9f2e4e to 0x74905c (0x2a9df1 bytes) 000170[1] 1788642590.068792 write 0x26f2de thru 0x270bc5 (0x18e8 bytes) 000171[0] 1788642590.125637 punch from 0x1fff11 to 0x2016d3 (0x17c3 bytes) 000172[0] 1788642590.165930 trunc from 0x74905d to 0x219b2b (0x52f531 bytes) 000176[0] 1788642590.258293 punch from 0x15a795 to 0x15ab70 (0x3dc bytes) 000177[0] 1788642590.325122 trunc from 0x219b2c to 0x1fc49a (0x1d691 bytes) 000178[1] 1788642590.384471 trunc from 0x1fc49b to 0x458c01 (0x25c767 bytes) 000180[0] 1788642590.525506 punch from 0x02c7d9 to 0x02e690 (0x1eb8 bytes) 000181[1] 1788642590.558185 write 0x51aaff thru 0x51b723 (0xc25 bytes) 000182[0] 1788642590.580130 trunc from 0x51b724 to 0x0e3b2c (0x437bf7 bytes) 000184[1] 1788642590.642015 read 0x0c0eea thru 0x0c2b05 (0x1c1c bytes) 000188[1] 1788642590.765148 mapwrite 0x212f6e thru 0x2133f9 (0x48c bytes) 000190[0] 1788642590.973915 trunc from 0x2133fa to 0x44377c (0x230383 bytes) 000191[0] 1788642591.075180 mapwrite 0x987c47 thru 0x988dbb (0x1175 bytes) 000192[0] 1788642591.265891 mapread 0x3cac73 thru 0x3cb180 (0x50e bytes) 000193[1] 1788642591.404181 read 0x46d402 thru 0x46eef4 (0x1af3 bytes) 000194[0] 1788642591.459496 punch from 0x2e0caf to 0x2e25d8 (0x192a bytes) 000195[0] 1788642591.470989 mapwrite 0x4c54bd thru 0x4c60ed (0xc31 bytes) 000196[1] 1788642591.660147 punch from 0x523ea3 to 0x525b6e (0x1ccc bytes) 000198[0] 1788642591.758892 mapwrite 0x5d916d thru 0x5db154 (0x1fe8 bytes) 000200[0] 1788642591.913741 mapwrite 0x1f7063 thru 0x1f7778 (0x716 bytes) 000202[1] 1788642591.973857 trunc from 0x988dbc to 0x2ffe77 (0x688f44 bytes) 000203[0] 1788642592.033893 read 0x28e221 thru 0x28e22e (0xe bytes) 000204[0] 1788642592.060027 mapread 0x130959 thru 0x13103c (0x6e4 bytes) 000207[1] 1788642592.081182 punch from 0x0e3d15 to 0x0e5008 (0x12f4 bytes) 000209[1] 1788642592.143378 write 0x7bee4c thru 0x7c0a32 (0x1be7 bytes) 000213[0] 1788642592.235019 mapwrite 0x9a8035 thru 0x9a8899 (0x865 bytes) 000217[1] 1788642592.527691 punch from 0x5aa314 to 0x5aa5dc (0x2c9 bytes) 000218[1] 1788642592.550677 read 0x90f4b1 thru 0x9101df (0xd2f bytes) 000221[0] 1788642592.625781 mapwrite 0x2da486 thru 0x2db256 (0xdd1 bytes) 000223[1] 1788642592.714184 punch from 0x35c42f to 0x35e261 (0x1e33 bytes) 000225[1] 1788642592.747484 read 0x63cb5c thru 0x63d86a (0xd0f bytes) 000226[0] 1788642592.778754 trunc from 0x9a889a to 0x6d048d (0x2d840c bytes) 000229[0] 1788642592.877113 trunc from 0x6d048e to 0x397530 (0x338f5d bytes) 000230[1] 1788642592.915650 trunc from 0x397531 to 0x50ceb1 (0x175981 bytes) 000231[1] 1788642592.977465 trunc from 0x50ceb2 to 0x4656aa (0xa7807 bytes) 000232[0] 1788642593.007652 write 0x8b7cef thru 0x8b9cc0 (0x1fd2 bytes) 000233[0] 1788642593.038740 mapread 0x5a5b20 thru 0x5a5dbc (0x29d bytes) 000235[0] 1788642593.082314 mapread 0x77da80 thru 0x77dbbf (0x140 bytes) 000236[1] 1788642593.088770 write 0x2f9448 thru 0x2f947e (0x37 bytes) 000237[1] 1788642593.121667 trunc from 0x8b9cc1 to 0x44e95a (0x46b366 bytes) 000239[0] 1788642593.191805 read 0x0f516e thru 0x0f6bba (0x1a4d bytes) 000243[1] 1788642593.260838 write 0x1d91ca thru 0x1d929d (0xd4 bytes) 000246[0] 1788642593.307913 mapread 0x275fed thru 0x276a26 (0xa3a bytes) 000250[0] 1788642593.360745 read 0x041488 thru 0x0422ad (0xe26 bytes) 000256[1] 1788642593.388467 trunc from 0x44e95b to 0x7041ac (0x2b5852 bytes) 000257[1] 1788642593.425711 punch from 0x4ca871 to 0x4cc35e (0x1aee bytes) 000258[1] 1788642593.455100 mapwrite 0x82dbfd thru 0x82fabc (0x1ec0 bytes) 000263[1] 1788642593.646788 mapwrite 0x28ffb0 thru 0x2916b0 (0x1701 bytes) 000264[0] 1788642593.721607 mapwrite 0x5d2db1 thru 0x5d4c9a (0x1eea bytes) 000265[0] 1788642593.813852 mapread 0x6a5a7a thru 0x6a6b1c (0x10a3 bytes) 000267[1] 1788642593.852418 mapwrite 0x12584f thru 0x127295 (0x1a47 bytes) 000268[0] 1788642593.929623 mapread 0x4bf22e thru 0x4c0ca7 (0x1a7a bytes) 000269[1] 1788642593.957564 trunc from 0x82fabd to 0x02d8b7 (0x802205 bytes) 000272[1] 1788642594.032754 mapwrite 0x511274 thru 0x513230 (0x1fbd bytes) 000273[1] 1788642594.110808 mapwrite 0x476625 thru 0x4773ae (0xd8a bytes) 000274[1] 1788642594.152654 mapwrite 0x54e40a thru 0x54efca (0xbc1 bytes) 000275[1] 1788642594.225062 trunc from 0x54efcb to 0x78f0bf (0x2400f5 bytes) 000276[0] 1788642594.262654 write 0x88cf29 thru 0x88ecfa (0x1dd2 bytes) 000278[1] 1788642594.317441 write 0x79c612 thru 0x79cdf3 (0x7e2 bytes) 000279[0] 1788642594.350430 mapread 0x830d09 thru 0x83110f (0x407 bytes) 000280[0] 1788642594.418077 trunc from 0x88ecfb to 0x34ff7e (0x53ed7c bytes) 000281[0] 1788642594.537510 mapread 0x07fd90 thru 0x080096 (0x307 bytes) 000283[0] 1788642594.603956 read 0x2698da thru 0x26a791 (0xeb8 bytes) 000284[0] 1788642594.764372 write 0x529290 thru 0x52af06 (0x1c77 bytes) 000285[1] 1788642594.767358 mapwrite 0x1ac50b thru 0x1ada29 (0x151f bytes) 000286[0] 1788642594.977923 mapread 0x1c18ef thru 0x1c3647 (0x1d59 bytes) 000289[0] 1788642595.145078 mapread 0x4ec4e4 thru 0x4ed6f7 (0x1214 bytes) 000291[0] 1788642595.234770 punch from 0x2e1ab7 to 0x2e35d7 (0x1b21 bytes) 000294[1] 1788642595.313603 read 0x1b21b4 thru 0x1b2aee (0x93b bytes) 000295[0] 1788642595.349693 write 0x6131d8 thru 0x613f6d (0xd96 bytes) 000297[1] 1788642595.371373 read 0x54ec07 thru 0x550465 (0x185f bytes) 000298[0] 1788642595.399462 trunc from 0x613f6e to 0x008a5b (0x60b512 bytes) 000300[1] 1788642595.465769 write 0x7b8d4e thru 0x7b98e4 (0xb97 bytes) 000303[1] 1788642595.522809 write 0x7fd038 thru 0x7fea49 (0x1a12 bytes) 000306[1] 1788642595.569452 trunc from 0x7fea4a to 0x792f59 (0x6baf0 bytes) 000307[1] 1788642595.598592 trunc from 0x792f5a to 0x0e22e0 (0x6b0c79 bytes) 000309[1] 1788642595.665046 read 0x0a5395 thru 0x0a5ed9 (0xb45 bytes) 000310[1] 1788642595.691647 write 0x70498d thru 0x705645 (0xcb9 bytes) 000311[0] 1788642595.714529 trunc from 0x705646 to 0x3b1002 (0x354643 bytes) 000313[1] 1788642595.764257 punch from 0x2050d3 to 0x206f5e (0x1e8c bytes) 000314[0] 1788642595.792535 write 0x6540b7 thru 0x654623 (0x56d bytes) 000317[1] 1788642595.838693 read 0x503d1d thru 0x504ea6 (0x118a bytes) 000318[0] 1788642595.861614 read 0x136081 thru 0x137370 (0x12f0 bytes) 000320[1] 1788642595.878808 trunc from 0x654624 to 0x5d4309 (0x8031a bytes) 000323[0] 1788642595.944482 read 0x1f6a19 thru 0x1f7d81 (0x1369 bytes) 000325[0] 1788642595.976764 mapwrite 0x418262 thru 0x418341 (0xe0 bytes) 000328[0] 1788642596.068450 read 0x19e6df thru 0x1a06bb (0x1fdd bytes) 000331[1] 1788642596.109488 mapwrite 0x289d03 thru 0x28ae4a (0x1148 bytes) 000333[0] 1788642596.257606 write 0x24e7d8 thru 0x24eff9 (0x822 bytes) 000335[0] 1788642596.306538 trunc from 0x5d430a to 0x972e56 (0x39eb4d bytes) 000336[1] 1788642596.361115 punch from 0x8103d8 to 0x8115cd (0x11f6 bytes) 000338[1] 1788642596.417833 mapread 0x015d9e thru 0x015e04 (0x67 bytes) 000340[1] 1788642596.480664 punch from 0x916b23 to 0x91750d (0x9eb bytes) 000341[1] 1788642596.510001 write 0x1438f7 thru 0x145261 (0x196b bytes) 000343[0] 1788642596.560477 punch from 0x071fc5 to 0x072edf (0xf1b bytes) 000347[0] 1788642596.664563 mapwrite 0x603627 thru 0x60476a (0x1144 bytes) 000350[0] 1788642596.733788 mapwrite 0x1db2cf thru 0x1dc56f (0x12a1 bytes) 000352[1] 1788642596.830250 mapread 0x26529f thru 0x26703c (0x1d9e bytes) 000353[1] 1788642596.889393 punch from 0x05b771 to 0x05bbab (0x43b bytes) 000354[1] 1788642596.928203 write 0x9ae561 thru 0x9af8d5 (0x1375 bytes) 000355[0] 1788642596.945394 read 0x2ffe43 thru 0x30043d (0x5fb bytes) 000356[0] 1788642596.998077 mapwrite 0x5f9612 thru 0x5fae65 (0x1854 bytes) 000357[0] 1788642597.059968 write 0x409b18 thru 0x40b46f (0x1958 bytes) 000358[0] 1788642597.091029 punch from 0x64d6fe to 0x64e43c (0xd3f bytes) 000361[0] 1788642597.134929 mapwrite 0x2b13ce thru 0x2b19d5 (0x608 bytes) 000362[1] 1788642597.232355 mapread 0x783d4f thru 0x784f13 (0x11c5 bytes) 000364[0] 1788642597.321386 mapread 0x36a2fd thru 0x36aae5 (0x7e9 bytes) 000366[1] 1788642597.345925 write 0x2e6d30 thru 0x2e7222 (0x4f3 bytes) 000368[1] 1788642597.375537 read 0x7d2185 thru 0x7d3bfe (0x1a7a bytes) 000369[1] 1788642597.385579 mapread 0x253cc2 thru 0x254e78 (0x11b7 bytes) 000372[0] 1788642597.396867 write 0x2d6fa7 thru 0x2d7b12 (0xb6c bytes) 000374[1] 1788642597.428287 mapwrite 0x2d626e thru 0x2d77f7 (0x158a bytes) 000376[1] 1788642597.540659 read 0x497f81 thru 0x498580 (0x600 bytes) 000377[0] 1788642597.561020 trunc from 0x9af8d6 to 0x624707 (0x38b1ce bytes) 000378[0] 1788642597.620041 write 0x3e59f3 thru 0x3e5ba8 (0x1b6 bytes) 000379[0] 1788642597.654738 punch from 0x4bd526 to 0x4beb9e (0x1679 bytes) 000380[0] 1788642597.662365 punch from 0x5d7a69 to 0x5d8034 (0x5cc bytes) 000381[0] 1788642597.674744 mapread 0x0d76c6 thru 0x0d8404 (0xd3f bytes) 000382[1] 1788642597.710992 mapwrite 0x4fe4cb thru 0x4ffed3 (0x1a09 bytes) 000384[0] 1788642597.929859 trunc from 0x624708 to 0x1d8134 (0x44c5d3 bytes) 000385[0] 1788642598.002887 read 0x024af5 thru 0x024d18 (0x224 bytes) 000389[1] 1788642598.201520 mapwrite 0x4f02a0 thru 0x4f1cc6 (0x1a27 bytes) 000392[0] 1788642598.355088 punch from 0x355695 to 0x3558eb (0x257 bytes) 000393[1] 1788642598.388927 mapread 0x291dcd thru 0x2938d2 (0x1b06 bytes) 000394[1] 1788642598.485317 mapwrite 0x24a1a9 thru 0x24b385 (0x11dd bytes) 000395[0] 1788642598.617943 mapwrite 0x4abddf thru 0x4abfcf (0x1f1 bytes) 000396[1] 1788642598.731523 write 0x8f8316 thru 0x8fa187 (0x1e72 bytes) 000397[0] 1788642598.745349 read 0x3f38b7 thru 0x3f4cd4 (0x141e bytes) 000398[1] 1788642598.780678 mapread 0x587332 thru 0x587e91 (0xb60 bytes) 000399[0] 1788642598.799589 mapread 0x3956fa thru 0x39650b (0xe12 bytes) 000400[0] 1788642598.812049 trunc from 0x8fa188 to 0x1e8b70 (0x711617 bytes) 000403[0] 1788642598.909594 read 0x1e7715 thru 0x1e820f (0xafb bytes) 000404[0] 1788642598.964058 punch from 0x1957a7 to 0x197378 (0x1bd2 bytes) 000407[1] 1788642598.995860 read 0x07a67a thru 0x07c664 (0x1feb bytes) 000408[0] 1788642599.047565 mapread 0x1b55dc thru 0x1b5ff2 (0xa17 bytes) 000409[1] 1788642599.067581 read 0x0bf4bb thru 0x0c0529 (0x106f bytes) 000410[0] 1788642599.068749 mapread 0x06b9eb thru 0x06d82c (0x1e42 bytes) 000411[0] 1788642599.083307 mapread 0x059187 thru 0x05b087 (0x1f01 bytes) 000413[1] 1788642599.096851 read 0x06c926 thru 0x06d2d7 (0x9b2 bytes) 000414[1] 1788642599.125204 mapread 0x0c2d71 thru 0x0c3ac0 (0xd50 bytes) 000417[0] 1788642599.222895 trunc from 0x1e8b71 to 0x27d86b (0x94cfb bytes) 000418[1] 1788642599.298589 mapread 0x1d558e thru 0x1d72c4 (0x1d37 bytes) 000419[0] 1788642599.418549 mapwrite 0x6dc704 thru 0x6dcda9 (0x6a6 bytes) 000420[0] 1788642599.704301 mapread 0x1a2d9b thru 0x1a431d (0x1583 bytes) 000421[1] 1788642599.787706 punch from 0x4d80d7 to 0x4d9698 (0x15c2 bytes) 000422[1] 1788642599.846879 punch from 0x04a568 to 0x04b173 (0xc0c bytes) 000423[1] 1788642599.883107 mapwrite 0x78a863 thru 0x78c254 (0x19f2 bytes) 000426[0] 1788642600.257356 punch from 0x601f7d to 0x603722 (0x17a6 bytes) 000427[1] 1788642600.370181 write 0x4efaeb thru 0x4f02ad (0x7c3 bytes) 000428[0] 1788642600.497123 trunc from 0x78c255 to 0x8e293f (0x1566eb bytes) 000430[1] 1788642600.685296 read 0x3ee4dd thru 0x3eea19 (0x53d bytes) 000431[1] 1788642600.714458 mapread 0x7f8301 thru 0x7f912a (0xe2a bytes) 000433[0] 1788642600.763806 mapread 0x10c05c thru 0x10c121 (0xc6 bytes) 000434[0] 1788642600.774150 mapread 0x17b18c thru 0x17ca7f (0x18f4 bytes) 000436[1] 1788642600.791185 read 0x6bc3a0 thru 0x6bd455 (0x10b6 bytes) 000437[1] 1788642600.815862 read 0x39a29c thru 0x39bdbf (0x1b24 bytes) 000438[1] 1788642600.823647 mapread 0x68f5cf thru 0x690bc7 (0x15f9 bytes) 000439[0] 1788642600.853125 read 0x83885a thru 0x83a780 (0x1f27 bytes) 000440[1] 1788642600.873684 mapwrite 0x5313e3 thru 0x5325f9 (0x1217 bytes) 000441[0] 1788642601.010122 mapwrite 0x907eb3 thru 0x9081d4 (0x322 bytes) 000443[1] 1788642601.162503 punch from 0x26f619 to 0x270f25 (0x190d bytes) 000445[0] 1788642601.222168 mapwrite 0x4b62a7 thru 0x4b6d0c (0xa66 bytes) 000447[0] 1788642601.328713 punch from 0x5f6640 to 0x5f7077 (0xa38 bytes) 000451[1] 1788642601.428691 read 0x776d00 thru 0x7783ad (0x16ae bytes) 000453[1] 1788642601.478497 mapread 0x76471c thru 0x765642 (0xf27 bytes) 000455[0] 1788642601.508430 read 0x75b71c thru 0x75b76f (0x54 bytes) 000456[0] 1788642601.558417 mapread 0x289aed thru 0x28a369 (0x87d bytes) 000457[1] 1788642601.596324 read 0x85ab44 thru 0x85ca9c (0x1f59 bytes) 000458[1] 1788642601.604972 punch from 0x236d2c to 0x238b11 (0x1de6 bytes) 000461[1] 1788642601.659760 mapread 0x02aa3f thru 0x02adbb (0x37d bytes) 000462[0] 1788642601.666398 write 0x95546b thru 0x956562 (0x10f8 bytes) 000464[0] 1788642601.689539 write 0x66b0ce thru 0x66c62d (0x1560 bytes) 000466[1] 1788642601.733425 punch from 0x20a8d7 to 0x20c490 (0x1bba bytes) 000471[0] 1788642601.819371 mapwrite 0x87ee56 thru 0x8808cb (0x1a76 bytes) 000472[1] 1788642601.953748 mapwrite 0x6a2399 thru 0x6a2ffd (0xc65 bytes) 000473[0] 1788642602.053049 mapread 0x261cb7 thru 0x262620 (0x96a bytes) 000474[1] 1788642602.091528 read 0x5716d2 thru 0x5730df (0x1a0e bytes) 000475[0] 1788642602.117155 write 0x674e04 thru 0x6764a9 (0x16a6 bytes) 000476[0] 1788642602.170000 mapwrite 0x7469e6 thru 0x747588 (0xba3 bytes) 000478[1] 1788642602.288929 mapread 0x45e00a thru 0x45f396 (0x138d bytes) 000479[0] 1788642602.321012 mapwrite 0x730461 thru 0x73240f (0x1faf bytes) 000480[0] 1788642602.439384 trunc from 0x956563 to 0x2b80f0 (0x69e472 bytes) 000481[1] 1788642602.472542 trunc from 0x2b80f1 to 0x0224ef (0x295c01 bytes) 000482[0] 1788642602.610576 mapread 0x00628f thru 0x00696d (0x6df bytes) 000483[0] 1788642602.664140 mapread 0x007783 thru 0x009268 (0x1ae6 bytes) 000491[0] 1788642602.797545 mapwrite 0x9fb62a thru 0x9fbcd2 (0x6a9 bytes) 000492[0] 1788642603.048990 write 0x9ac208 thru 0x9ac61a (0x413 bytes) 000494[1] 1788642603.104153 write 0x73a3fc thru 0x73c2f0 (0x1ef5 bytes) 000497[0] 1788642603.174595 trunc from 0x9fbcd3 to 0x73e3ce (0x2bd904 bytes) 000500[1] 1788642603.313337 mapwrite 0x8bc9e9 thru 0x8bd879 (0xe91 bytes) All operations completed A-OK! PASS 16b (30s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 17:10:08 (1788642608) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 17:10:13 (1788642613) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.39485 s, 21.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.268654 s, 31.2 MB/s ldlm.namespaces.MGC192.168.204.111@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9e9fc58a7800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9e9fc6e7a800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc6e7a800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc6e7a800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9071 s, 9.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.04793 s, 34.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.02589 s, 51.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23437 s, 20.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.2133 s, 12.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.6863 s, 22.4 MB/s PASS 16d (46s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 17:10:59 (1788642659) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.451411 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124567 s, 8.4 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 17:11:07 (1788642667) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 17:11:35 (1788642695) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 17:12:03 (1788642723) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 17:12:10 (1788642730) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 17:12:17 (1788642737) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.04074 s, 25.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.42092 s, 18.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.52978 s, 10.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.63033 s, 16.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.12278 s, 12.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.87512 s, 14.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.75446 s, 9.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.59056 s, 10.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.64242 s, 9.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.52237 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.67658 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.30866 s, 11.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.57281 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.96507 s, 8.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.10398 s, 8.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.58765 s, 10.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.89313 s, 9.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.38172 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.68267 s, 7.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (36s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 17:12:53 (1788642773) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: skipping zero size read Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Sat Sep 5 17:13:38 EDT 2026 + fsx 2 done: rc=0 Sat Sep 5 17:13:38 EDT 2026 + fsx 3 done: rc=0 Sat Sep 5 17:13:38 EDT 2026 + fsx 4 done: rc=0 Sat Sep 5 17:13:38 EDT 2026 + fsx 5 done: rc=0 Sat Sep 5 17:13:38 EDT 2026 PASS 16k (53s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 17:13:46 (1788642826) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 17:13:57 (1788642837) mmap test1: basic mmap operation (PASS, 0.13743s) mmap test2: MAP_PRIVATE not write back (PASS, 0.099935s) mmap test3: concurrent mmap ops on two nodes (PASS, 13.172s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 106.65s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.12879s) mmap test6: check mmap write/read content on two nodes (PASS, 0.33987s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.4833s) mmap test11: multi-thread shared mmap (PASS, 1.3618s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1473s) PASS 18 (134s) == sanityn test 19: test concurrent uncached read races ========================================================================= 17:16:11 (1788642971) oleg411-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 17:16:15 (1788642975) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 17:16:23 (1788642983) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 17:16:30 (1788642990) now is 1788642993 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 new atime is 1788643054 PASS 23 (69s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 17:17:40 (1788643060) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 17:17:46 (1788643066) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 17:17:53 (1788643073) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 17:18:00 (1788643080) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 17:18:03 (1788643083) utime: good mknod times 1788643084 <= 1788643085 <= 1788643085 for /mnt/lustre/f26a utime: good mknod times 1788643084 <= 1788643085 <= 1788643085 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 17:18:11 (1788643091) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 17:18:20 (1788643100) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-05 17:18:22.000000000 -0400 Birth: 2026-09-05 17:18:22.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-05 17:18:22.000000000 -0400 Birth: 2026-09-05 17:18:22.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 17:18:30 (1788643110) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 17:18:40 (1788643120) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 17:18:48 (1788643128) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg411-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.111@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 17:19:09 (1788643149) -rw-r--r-- 1 root root 0 Sep 5 17:19 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 17:19:19 (1788643159) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43209 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 17:19:26 (1788643166) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 17:19:35 (1788643175) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 17:19:38 (1788643178) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.42097 s, 36.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.10201 s, 47.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.56521 s, 33.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.71168 s, 30.6 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.69564 s, 30.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.82875 s, 28.7 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.19528 s, 43.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.20709 s, 43.4 MB/s PASS 32c (25s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 17:20:05 (1788643205) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 17:20:09 (1788643209) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 17:20:12 (1788643212) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 17:20:15 (1788643215) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 17:20:19 (1788643219) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e9fc58a7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e9fc58a7800.ost_server_uuid in IDLE state after 0 sec oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e9fc58a7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e9fc58a7800.ost_server_uuid in IDLE state after 0 sec oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e9fc58a7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e9fc58a7800.ost_server_uuid in IDLE state after 0 sec oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e9fc58a7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e9fc58a7800.ost_server_uuid in IDLE state after 0 sec oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e9fc58a7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e9fc58a7800.ost_server_uuid in IDLE state after 0 sec oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e9fc58a7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e9fc58a7800.ost_server_uuid in IDLE state after 0 sec PASS 34 (132s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 17:22:31 (1788643351) fail_loc=0x80000317 Race attempt 0 Wait for 50503 50572 for 60 sec... - open/close 768 (time 1788643366.49 total 10.01 last 76.75) - open/close 1518 (time 1788643376.51 total 20.03 last 74.79) - open/close 2233 (time 1788643386.53 total 30.05 last 71.35) - open/close 2961 (time 1788643396.54 total 40.06 last 72.78) - open/close 3746 (time 1788643406.55 total 50.07 last 78.38) total: 4000 open/close in 53.24 seconds: 75.13 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.111@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.111@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (70s) == sanityn test 36: handle ESTALE/open-unlink correctly == 17:23:41 (1788643421) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg411-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35213 s, 38.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.24774 s, 42.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7491584) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.996104 s, 52.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7495680) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.16848 s, 44.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.82274 s, 28.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7490560) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.71629 s, 30.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.82132 s, 28.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.13944 s, 24.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7444480) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.51807 s, 34.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49232 s, 35.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33621 s, 39.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7491584) after_dd(7446528) after(7497728) PASS 36 (337s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 17:29:18 (1788643758) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 - create 1316 (time 1788643772.57 total 10.00 last 131.59) - create 2671 (time 1788643782.59 total 20.01 last 135.32) - create 4060 (time 1788643792.59 total 30.02 last 138.87) - create 5448 (time 1788643802.59 total 40.02 last 138.71) - create 6784 (time 1788643812.60 total 50.03 last 133.50) - create 8186 (time 1788643822.61 total 60.04 last 140.11) - create 9743 (time 1788643832.62 total 70.05 last 155.55) total: 10000 create in 71.66 seconds: 139.56 ops/second PASS 37 (82s) == sanityn test 39a: file mtime does not change after rename ========================================================== 17:30:40 (1788643840) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 17:30:48 (1788643848) repeat after cancel_lru_locks PASS 39b (10s) == sanityn test 39c: check truncate mtime update ================================================================================ 17:30:57 (1788643857) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 17:31:06 (1788643866) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 17:31:15 (1788643875) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (19s) == sanityn test 40c: pdirops: link and others ======================================================================== 17:31:34 (1788643894) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 17:31:51 (1788643911) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (16s) == sanityn test 40e: pdirops: rename and others ======================================================================== 17:32:07 (1788643927) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 17:32:23 (1788643943) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 17:32:35 (1788643955) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 17:32:48 (1788643968) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 17:33:02 (1788643982) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 17:33:17 (1788643997) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 17:33:30 (1788644010) Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 17:33:43 (1788644023) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 17:33:57 (1788644037) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (15s) == sanityn test 41i: reint_open: create vs create ======== 17:34:12 (1788644052) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1071s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 17:52:04 (1788645124) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 17:52:17 (1788645137) open: File exists Conflict PASS 42b (14s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 17:52:31 (1788645151) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (12s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 17:52:43 (1788645163) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 17:52:56 (1788645176) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 17:53:09 (1788645189) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 17:53:23 (1788645203) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 17:53:37 (1788645217) Conflict PASS 42h (12s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 17:53:49 (1788645229) PASS 43a (70s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 17:54:59 (1788645299) Conflict PASS 43b (13s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 17:55:12 (1788645312) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 17:55:26 (1788645326) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 17:55:39 (1788645339) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 17:55:52 (1788645352) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (11s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 17:56:04 (1788645364) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (11s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 17:56:15 (1788645375) Conflict PASS 43h (12s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 17:56:27 (1788645387) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 17:56:31 (1788645391) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (115s) == sanityn test 43k: unlink vs create ==================== 17:58:26 (1788645506) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1138s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 18:17:24 (1788646644) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sat Sep 5 18:17:32 EDT 2026 PASS 44a (12s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 18:17:36 (1788646656) open: File exists Conflict PASS 44b (13s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 18:17:49 (1788646669) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (11s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 18:18:00 (1788646680) Conflict PASS 44d (11s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 18:18:12 (1788646692) Conflict PASS 44e (12s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 18:18:23 (1788646703) Conflict PASS 44f (12s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 18:18:35 (1788646715) Conflict PASS 44g (13s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 18:18:48 (1788646728) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 18:19:01 (1788646741) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 18:19:05 (1788646745) PASS 45a (119s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 18:21:04 (1788646864) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 18:21:17 (1788646877) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 18:21:30 (1788646890) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 18:21:44 (1788646904) Conflict PASS 45e (13s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 18:21:57 (1788646917) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (11s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 18:22:08 (1788646928) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (10s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 18:22:18 (1788646938) Conflict PASS 45h (12s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 18:22:30 (1788646950) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 18:22:33 (1788646953) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1103s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 18:40:56 (1788648056) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (14s) == sanityn test 46b: pdirops: link vs create ======================================================================== 18:41:11 (1788648071) open: File exists Conflict PASS 46b (14s) == sanityn test 46c: pdirops: link vs link ======================================================================== 18:41:25 (1788648085) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 18:41:39 (1788648099) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 18:41:52 (1788648112) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 18:42:06 (1788648126) Conflict PASS 46f (11s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 18:42:18 (1788648138) Conflict PASS 46g (12s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 18:42:30 (1788648150) Conflict PASS 46h (11s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 18:42:41 (1788648161) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 18:42:44 (1788648164) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 18:42:48 (1788648168) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (5s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 18:42:53 (1788648173) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 18:42:57 (1788648177) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 18:43:01 (1788648181) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 18:43:04 (1788648184) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 18:43:08 (1788648188) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 18:43:12 (1788648192) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.154455 s, 66.3 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 18:43:25 (1788648205) 0+1 records in 0+1 records out 158 bytes copied, 0.0221914 s, 7.1 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218851) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 18:43:33 (1788648213) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0130186 s, 78.7 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805012 s, 13.0 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 18:43:54 (1788648234) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0226523 s, 45.2 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 18:44:06 (1788648246) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648686 s, 16.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (7s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 18:44:14 (1788648254) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (7s) == sanityn test 54: rename locking ======================= 18:44:22 (1788648262) llite.lustre-ffff9e9fc58a7800.xattr_cache=1 llite.lustre-ffff9e9fc6e7a800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 18:44:23.000000000 -0400 Modify: 2026-09-05 18:44:23.000000000 -0400 Change: 2026-09-05 18:44:23.000000000 -0400 Birth: 2026-09-05 18:44:23.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 18:44:29.000000000 -0400 Modify: 2026-09-05 18:44:30.000000000 -0400 Change: 2026-09-05 18:44:30.000000000 -0400 Birth: 2026-09-05 18:44:30.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 18:44:35.000000000 -0400 Modify: 2026-09-05 18:44:36.000000000 -0400 Change: 2026-09-05 18:44:36.000000000 -0400 Birth: 2026-09-05 18:44:36.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 18:44:35.000000000 -0400 Modify: 2026-09-05 18:44:37.000000000 -0400 Change: 2026-09-05 18:44:37.000000000 -0400 Birth: 2026-09-05 18:44:36.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 18:44:42.000000000 -0400 Modify: 2026-09-05 18:44:43.000000000 -0400 Change: 2026-09-05 18:44:43.000000000 -0400 Birth: 2026-09-05 18:44:42.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 18:44:42.000000000 -0400 Modify: 2026-09-05 18:44:43.000000000 -0400 Change: 2026-09-05 18:44:43.000000000 -0400 Birth: 2026-09-05 18:44:42.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (31s) == sanityn test 55a: rename vs unlink target dir ========= 18:44:53 (1788648293) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (11s) == sanityn test 55b: rename vs unlink source dir ========= 18:45:05 (1788648305) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 18:45:18 (1788648318) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 18:45:37 (1788648337) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 18:45:52 (1788648352) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 18:45:55 (1788648355) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (18s) == sanityn test 55g: rename: race with trylock =========== 18:46:13 (1788648373) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (20s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 18:46:34 (1788648394) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788648395 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.701598 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (96s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 18:48:09 (1788648489) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788648495 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.19681 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (115s) == sanityn test 60: Verify data_version behaviour ======== 18:50:04 (1788648604) PASS 60 (8s) == sanityn test 70a: cd directory && rm directory ======== 18:50:12 (1788648612) PASS 70a (8s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 18:50:20 (1788648620) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 18:50:29 (1788648629) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0199004 s, 3.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 18:50:32 (1788648632) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0196721 s, 3.3 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 18:50:35 (1788648635) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 18:50:38 (1788648638) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.025082 s, 2.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 18:50:41 (1788648641) llite.lustre-ffff9e9fc58a7800.xattr_cache=1 llite.lustre-ffff9e9fc6e7a800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 18:50:47 (1788648647) llite.lustre-ffff9e9fc58a7800.xattr_cache=1 llite.lustre-ffff9e9fc6e7a800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (5s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 18:50:52 (1788648652) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0130207 s, 78.6 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.014282 s, 71.7 kB/s 231647: get lock2 231651: get lock1 231647: done 231647: get lock1 231651: done 231651: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231647: cannot get lock1: Resource deadlock avoided 231651: done 231651: exit rc=0 231647: done 231647: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 18:51:03 (1788648663) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0893785 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408163 s, 25.7 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232232 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 18:51:15 (1788648675) opening files: ........ closing files: ........ PASS 76 (71s) == sanityn test 77a: check FIFO NRS policy =============== 18:52:26 (1788648746) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.36468 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110313 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116797 s, 9.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672153 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12486 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102643 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107705 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973137 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206927 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200643 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173565 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175352 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131368 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205107 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203102 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215107 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254638 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460596 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042384 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552696 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435214 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416297 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506175 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054866 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503092 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534249 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053636 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580074 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560846 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536652 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532739 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00642833 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639645 s, 16.4 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 18:52:35 (1788648755) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.489442 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802871 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069412 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563822 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972729 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115309 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111193 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723633 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196583 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162194 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176545 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199253 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251634 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187056 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214806 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218999 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292759 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683083 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229816 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255822 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302809 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434982 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311967 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329934 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10245 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480565 s, 21.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480725 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537394 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538955 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058197 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607619 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632762 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673573 s, 15.6 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.475203 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772112 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782163 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129595 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158175 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192072 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295002 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241409 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254956 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252804 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340767 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279371 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37323 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333919 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350352 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364147 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491689 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00512288 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308585 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138281 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00617194 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364816 s, 28.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=10 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189071 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18746 s, 5.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964095 s, 10.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00724304 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168181 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206076 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197618 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360097 s, 29.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289168 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208123 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180276 s, 5.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (13s) == sanityn test 77c: check ORR NRS policy ================ 18:52:48 (1788648768) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.365774 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698972 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893499 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215219 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187046 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2344 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245157 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242723 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245253 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283049 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313332 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303441 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319478 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313365 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332268 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246599 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355048 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705358 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120711 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135348 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876878 s, 12.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125197 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152075 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105408 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637075 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681909 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607968 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506982 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440017 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573977 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415003 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388555 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842741 s, 12.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.522813 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131068 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915653 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123337 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150001 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129072 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248882 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144673 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109064 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240457 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100206 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230886 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328505 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201924 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346114 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339065 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195283 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399931 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379665 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195048 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269745 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039627 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713095 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343902 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350534 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322088 s, 32.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384547 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934815 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00587292 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318389 s, 32.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186005 s, 56.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016347 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00753672 s, 139 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.594741 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554377 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610069 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132961 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174167 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150213 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10227 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209858 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264936 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291534 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228764 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290432 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383058 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283291 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420899 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375846 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342485 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972052 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244387 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105541 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535802 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607526 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665792 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725246 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755986 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816841 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912149 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882008 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100542 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120533 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107405 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183764 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125786 s, 8.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (16s) == sanityn test 77d: check TRR nrs policy ================ 18:53:05 (1788648785) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.413371 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111071 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116551 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110229 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197939 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275146 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329965 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24768 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339624 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331559 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205309 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329894 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282612 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308511 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391885 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.498894 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482259 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681694 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471868 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121437 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0059641 s, 176 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142428 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158708 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144613 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145761 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487763 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847259 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798739 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195214 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072775 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732108 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836964 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765542 s, 13.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.262326 s, 64.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635718 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927749 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100917 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968683 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175027 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150524 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15816 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179683 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258456 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274786 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217263 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303753 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335788 s, 3.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22068 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259555 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269004 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831946 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636407 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641275 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974621 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851999 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768675 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972921 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106018 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869769 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139963 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11513 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997413 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200042 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126611 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130665 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133913 s, 7.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.525781 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400935 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458209 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722956 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580616 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113427 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761274 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111507 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200379 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183721 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18886 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22782 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195888 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245097 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241281 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260209 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264455 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036382 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583854 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00267972 s, 391 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226532 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039847 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228666 s, 459 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603791 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599964 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981805 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924886 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429388 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318698 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058523 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131424 s, 79.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208189 s, 50.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249808 s, 42.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (16s) == sanityn test 77e: check TBF NID nrs policy ============ 18:53:21 (1788648801) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.11@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.44756 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026224 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613554 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128342 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114006 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133035 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592995 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138338 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247143 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162254 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230431 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227789 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174065 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235539 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280505 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244378 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328931 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131644 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086757 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135297 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1243 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806001 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557436 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128366 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766655 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474292 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733253 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397842 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012466 s, 84.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443007 s, 23.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244501 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045367 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323764 s, 32.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.434844 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369982 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530986 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942118 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948481 s, 11.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151079 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172455 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160705 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17484 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166588 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192326 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182669 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201506 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19972 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209862 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199403 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212446 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152518 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12771 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134434 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134997 s, 7.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135953 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137671 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173163 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14302 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16236 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149658 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134853 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163071 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140514 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155473 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142101 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151314 s, 6.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.53372 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899937 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958484 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892729 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2373 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213499 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232325 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307691 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300273 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340278 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30625 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254685 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348786 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275315 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368512 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337955 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317307 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00310833 s, 337 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00347764 s, 302 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906128 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339771 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106515 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709993 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00823898 s, 127 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00625279 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104104 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00702372 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756648 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290494 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087734 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033802 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287742 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00841492 s, 125 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.61736 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050807 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163107 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120815 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165785 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205168 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185392 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203191 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270036 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178115 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247269 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302521 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273383 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308583 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254664 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316658 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408198 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762237 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651027 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143018 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163868 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035272 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869137 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180694 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813049 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129702 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19514 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843313 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109014 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245837 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222363 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108037 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00899877 s, 117 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (24s) == sanityn test 77f: check TBF JobID nrs policy ========== 18:53:45 (1788648825) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.662438 s, 25.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121061 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852185 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106629 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117642 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219547 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238312 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333633 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250912 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295323 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281188 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331331 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21593 s, 4.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404425 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413759 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147914 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471479 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945688 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994667 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532574 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654214 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508207 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653744 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709171 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647446 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13272 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604111 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818109 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784051 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779651 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929891 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807992 s, 13.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103241 s, 10.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.278687 s, 60.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405747 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128158 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108936 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258834 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208713 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830808 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126121 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295654 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411731 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4095 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409639 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403431 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384808 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481854 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361089 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.526608 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769833 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216231 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00640924 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130081 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722807 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794287 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225983 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156257 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944854 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097875 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110592 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104682 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112965 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118749 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124022 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137057 s, 7.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.449769 s, 37.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571506 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115525 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233098 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200289 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297009 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235955 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265122 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297976 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304883 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325146 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293144 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355747 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339079 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360256 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323497 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495336 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704978 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843459 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518079 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176563 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815713 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734822 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455626 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526313 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00780081 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425919 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401291 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361129 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339481 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029566 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027034 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237192 s, 44.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.668183 s, 25.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160417 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217524 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246219 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303235 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325912 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296925 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369294 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337536 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367557 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358932 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376905 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415452 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.600497 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493076 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440063 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.536715 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132663 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659237 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00318145 s, 330 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144949 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013889 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00902769 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171547 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12356 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157276 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122973 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101848 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155893 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834078 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122498 s, 85.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052219 s, 20.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373944 s, 28.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (28s) == sanityn test 77g: Change TBF type directly ============ 18:54:13 (1788648853) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 18:54:24 (1788648864) oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg411-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg411-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg411-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg411-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 PASS 77h (11s) == sanityn test 77i: Change rank of TBF rule ============= 18:54:35 (1788648875) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (21s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 18:54:56 (1788648896) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.505012 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298406 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428221 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701957 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099398 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951915 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077119 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107723 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125131 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148826 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167315 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181511 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156246 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181889 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145942 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195396 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199173 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245299 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696373 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614581 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374726 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06805 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577038 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767904 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038985 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087758 s, 11.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742074 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753886 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649074 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046553 s, 22.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703657 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057997 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630858 s, 16.6 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.04982 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6117 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (65s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 18:56:01 (1788648961) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg411-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg411-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg411-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg411-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31816 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572086 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056984 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942302 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155726 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204145 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220819 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187176 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184912 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200415 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230987 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28319 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277516 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33345 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329715 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320461 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351994 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364189 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593427 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418877 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555289 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651059 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720651 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749254 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693172 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079367 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595604 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864561 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910802 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938935 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694423 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628829 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973443 s, 10.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2616 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9484 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg411-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg411-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg411-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg411-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25315 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100942 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117179 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127673 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13581 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123832 s, 8.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127711 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139605 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130329 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132943 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130262 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168804 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183385 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213528 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258302 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243081 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296136 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712693 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727066 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757311 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771801 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073398 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794582 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747779 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923203 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933043 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855281 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919984 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105906 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102551 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102372 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100622 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109549 s, 9.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0699 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9181 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (153s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 18:58:34 (1788649114) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg411-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg411-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg411-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg411-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28981 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921158 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958858 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114581 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106635 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099611 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103483 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958949 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173415 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196438 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116006 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142614 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227094 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128638 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180358 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209328 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269637 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476835 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449664 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059328 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362977 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510621 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747056 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412323 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072436 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550997 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645549 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616775 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631243 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681292 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919506 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723896 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837524 s, 12.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9841 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9481 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg411-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg411-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg411-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg411-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.34067 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810689 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885316 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117016 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106018 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204264 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117389 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215546 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243457 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249418 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250264 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233538 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241177 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259965 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282534 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248367 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267102 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602897 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683417 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537594 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070364 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084081 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644215 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857109 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508161 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886086 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786517 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953744 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104163 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906743 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779407 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948645 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125627 s, 8.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9137 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.951 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (153s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 19:01:07 (1788649267) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.953859 s, 17.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675052 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146722 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870947 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148116 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23539 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188848 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309089 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28603 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27292 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373519 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261287 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28553 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400006 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20401 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36411 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407199 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966251 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322457 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00252137 s, 416 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569405 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00330238 s, 318 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121323 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782579 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09536 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507825 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045892 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407401 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157784 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353694 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305819 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245268 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219278 s, 47.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.51123 s, 11.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92766 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.11@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.11@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.541396 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671779 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719339 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634982 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791914 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859874 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951775 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239008 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211299 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287378 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265792 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253455 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291118 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311218 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428397 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45596 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418373 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169859 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142915 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212408 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143849 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1582 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193108 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160872 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173366 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174978 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176985 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184476 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184072 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167767 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261085 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222256 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193804 s, 5.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.43212 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89938 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.11@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.582357 s, 28.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084335 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109342 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060149 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157863 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178501 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191034 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178206 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251468 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245792 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259651 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320657 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326671 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327569 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3205 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341719 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc58a7800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40194 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363925 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706346 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372139 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542726 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487633 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420782 s, 24.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764933 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393145 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00220787 s, 475 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103206 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426325 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644011 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796087 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539999 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408238 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626347 s, 16.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.47413 s, 14.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.2841 s, 19.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.615516 s, 27.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357481 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697863 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929992 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102192 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138586 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104837 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308231 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250818 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330228 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214594 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32707 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265663 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312357 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284762 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352818 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401325 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384769 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788819 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812781 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740574 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615558 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792357 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519804 s, 20.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121745 s, 8.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850987 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100129 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912627 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859286 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842688 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717739 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143557 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113155 s, 9.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7347 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9295 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.45088 s, 11.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88981 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.950939 s, 17.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958659 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689349 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118564 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649522 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127404 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226731 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620545 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175746 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994415 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282289 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207695 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234508 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23386 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257683 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248038 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223522 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156411 s, 67.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579395 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158683 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017189 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298396 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021934 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214056 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024166 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261986 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340549 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285883 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310353 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363658 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373037 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465293 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510718 s, 20.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7641 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5709 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.498068 s, 33.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305675 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574538 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672591 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732078 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773063 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082331 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101768 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610097 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912273 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160126 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17461 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220339 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203048 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215228 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199932 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240801 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101664 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591937 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246364 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109977 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109738 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530051 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037068 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498678 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101145 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653763 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854166 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072917 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935667 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00697013 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00466872 s, 225 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110851 s, 9.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.75839 s, 13.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97774 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.633554 s, 26.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303919 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740555 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789776 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152805 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145808 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16841 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176258 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162844 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201374 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230701 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163423 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157683 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214908 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243192 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176722 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201282 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076359 s, 13.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06329 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599975 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700767 s, 15.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530407 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726156 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644491 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683223 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100822 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822303 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726875 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100731 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727727 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743289 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084114 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089806 s, 11.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7459 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90158 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4659 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92683 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (424s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 19:08:12 (1788649692) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.473502 s, 35.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071696 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598618 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487393 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764926 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314058 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616274 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100316 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576521 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086711 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506288 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211904 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210902 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174731 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258084 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206571 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624062 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678272 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343335 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147225 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763548 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278535 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110287 s, 95.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738237 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00416932 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787319 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878575 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353653 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526433 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00323925 s, 324 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337896 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476779 s, 22.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.70833 s, 15.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95203 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (48s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 19:09:00 (1788649740) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.11@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.11@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.619372 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514234 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627676 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426075 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299734 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633643 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659025 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095494 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076823 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771489 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967492 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116843 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135294 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185885 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169112 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144707 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16314 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398634 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986902 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070164 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466763 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496316 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866161 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133533 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617229 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968848 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945216 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705868 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081315 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832567 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952402 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886552 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997183 s, 10.5 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.47872 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94398 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (53s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 19:09:53 (1788649793) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.11@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.532887 s, 31.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462269 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437712 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638547 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209165 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196816 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269144 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251471 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262499 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307975 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30836 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314236 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310879 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316742 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280874 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296131 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347504 s, 3.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245497 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490427 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202946 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058161 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486344 s, 21.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044402 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631587 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610318 s, 17.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043497 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514659 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842657 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615787 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580226 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539149 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490305 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295699 s, 35.5 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.46197 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.15183 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (44s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 19:10:37 (1788649837) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.557252 s, 30.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422527 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536414 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147119 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152517 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16684 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162544 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186315 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174536 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194835 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208363 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18199 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201066 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184236 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211739 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236224 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21501 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408874 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950807 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310107 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103273 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00734852 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635586 s, 16.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588395 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1183 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320896 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923485 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108074 s, 9.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895096 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103517 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115651 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123555 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275755 s, 38.0 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5009 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9171 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9194 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0636 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (102s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 19:12:19 (1788649939) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.954355 s, 17.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447478 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900349 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124592 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147558 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131937 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251347 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242386 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21742 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178921 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282917 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258787 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248744 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25013 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27873 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307983 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23353 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119415 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986716 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645047 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161276 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16783 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132092 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150545 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12041 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614316 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975097 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570919 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508079 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526016 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580084 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456494 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469817 s, 22.3 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1006 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9389 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (76s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 19:13:35 (1788650015) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.537239 s, 31.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493345 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699427 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757681 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194898 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212076 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211938 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23791 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282699 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241357 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253761 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295719 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305873 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303677 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324865 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321025 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350881 s, 3.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc58a7800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100716 s, 10.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115132 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677279 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107681 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647392 s, 16.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639022 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124709 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115811 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14166 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109244 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856008 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963905 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153075 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016059 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818025 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271548 s, 38.6 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.88266 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96356 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg411-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.775167 s, 21.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293489 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118754 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154369 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176019 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206458 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110608 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140553 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164626 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167924 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291588 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186745 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228046 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233031 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238011 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310538 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291687 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420105 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382412 s, 274 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827663 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519273 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971186 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289281 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105085 s, 99.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811539 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849568 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909045 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841954 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014224 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163445 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00754788 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00878951 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150998 s, 6.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.909 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96049 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9332 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0655 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (146s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 19:16:01 (1788650161) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg411-client.virtnet /mnt/lustre (opts:) Stopping client oleg411-client.virtnet /mnt/lustre opts: Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg411-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg411-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg411-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.11,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg411-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.11,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg411-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.11,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69617/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.11,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg411-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: kmemleak disabled oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre Starting client oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre Started clients oleg411-client.virtnet: 192.168.204.111@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 Starting client oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 Started clients oleg411-client.virtnet: 192.168.204.111@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e9fc4990000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e9fc58c1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e9fc4990000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e9fc58c1800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.441423 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756157 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106676 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206845 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283221 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240216 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282177 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268845 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285446 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331234 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292089 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359212 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335647 s, 3.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330878 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384207 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405796 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356773 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00657897 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022095 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483279 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054635 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333064 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378042 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890006 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239597 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496033 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978655 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00741709 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563729 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00985678 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00947652 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116036 s, 90.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147347 s, 71.2 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.23573 s, 11.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.18627 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.629435 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807562 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873868 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902359 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199143 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225708 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239172 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173618 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254272 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287451 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231284 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25455 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207134 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265133 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284235 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228843 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267775 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc4990000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463855 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732858 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378756 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421117 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414432 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043589 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472203 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482479 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498725 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619625 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517297 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582806 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578636 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063943 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662553 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750308 s, 14.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.07926 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92674 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (229s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 19:19:51 (1788650391) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg411-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 19:20:10 (1788650410) On MGS 192.168.204.111, active = nodemap.active=1 On MGS 192.168.204.111, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.76769 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136009 s, 7.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187381 s, 5.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161735 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205444 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204967 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140432 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288134 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271684 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314893 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175209 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306536 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187585 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238213 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31807 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282932 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244206 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480955 s, 2.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc4990000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398024 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373715 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364565 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426018 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386102 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415054 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348154 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449319 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.467587 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389909 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395911 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355597 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.543757 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356811 s, 2.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372375 s, 2.8 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.49498 s, 12.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.24044 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.111, TBF.id = On MGS 192.168.204.111, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.662357 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932706 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118721 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187263 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288703 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276952 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275911 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286654 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41444 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351446 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368313 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397106 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466801 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365577 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457333 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.558312 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.504192 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110414 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110867 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417826 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662249 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570761 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743031 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138204 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821779 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129638 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912735 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117177 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109477 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954831 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171452 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182031 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108343 s, 9.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc4990000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc4990000.lock_unused_count=6 removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.16953 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.7863 s, 37.6 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg411-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg411-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 On MGS 192.168.204.111, active = nodemap.active=1 On MGS 192.168.204.111, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.560867 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102265 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100016 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760674 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104422 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139032 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260848 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287303 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282427 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316934 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246187 s, 4.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278212 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269829 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272693 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342224 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340438 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344426 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346712 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550416 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266421 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558187 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577781 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570479 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774482 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697235 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631108 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553879 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740548 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658359 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783177 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714537 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912427 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839143 s, 12.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.25881 s, 12.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95531 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.111, TBF.id = On MGS 192.168.204.111, active = nodemap.active=0 PASS 77kj (191s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 19:23:21 (1788650601) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 19:23:32 (1788650612) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg411-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.72015 s, 282 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg411-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1199 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 19:24:31 (1788650671) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.532862 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585536 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980066 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150565 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158916 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144519 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130751 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187212 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12454 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231394 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288468 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305763 s, 3.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc4990000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210868 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240065 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247734 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339133 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287185 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc4990000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221777 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00807243 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302875 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068546 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10065 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00276853 s, 379 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411913 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586878 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386787 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497425 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434557 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385266 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635761 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625832 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169094 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685768 s, 15.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg411-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg411-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.45999 s, 11.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20513 s, 20.1 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.822493 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869708 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723121 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439461 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573876 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036965 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116879 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108681 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109414 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423112 s, 24.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9e9fc4990000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177735 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22966 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196524 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245027 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317876 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348274 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404255 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9e9fc4990000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457199 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.526938 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445781 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450093 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471403 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453312 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438726 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449502 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430365 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52658 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482748 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.548251 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.520749 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493006 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487644 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459007 s, 2.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg411-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg411-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.57473 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.22216 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (94s) == sanityn test 77o: Changing rank should not panic ====== 19:26:05 (1788650765) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 19:26:18 (1788650778) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (116s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 19:28:14 (1788650894) mds.MDS.mdt.nrs_policies=tbf oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg411-client: oleg411-server: ssh exited with exit code 36 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 oleg411-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg411-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg411-client: oleg411-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (38s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 19:28:53 (1788650933) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273946 kill md_thread xargs PASS 77r (53s) == sanityn test 77s: Check TBF LRU shrinker ============== 19:29:46 (1788650986) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (66s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 19:30:53 (1788651053) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 19:31:01 (1788651061) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935727 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 19:31:03.000000000 -0400 Modify: 2026-09-05 19:31:03.000000000 -0400 Change: 2026-09-05 19:31:03.000000000 -0400 Birth: 2026-09-05 19:31:03.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (18s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 19:31:20 (1788651080) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 19:31:23 (1788651083) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 19:31:27 (1788651087) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 19:31:30 (1788651090) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 19:31:34 (1788651094) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 19:31:37 (1788651097) - open/close 680 (time 1788651112.69 total 10.01 last 67.94) - open/close 1409 (time 1788651122.71 total 20.03 last 72.76) - open/close 2182 (time 1788651132.72 total 30.03 last 77.26) total: 2500 open/close in 34.94 seconds: 71.55 ops/second snapshot_time 1788651225.056201254 secs.nsecs start_time 1788651100.386438947 secs.nsecs elapsed_time 124.669762307 secs.nsecs open 2505 samples [usecs] 181 18983 4219702 12823855916 close 2505 samples [usecs] 159 11496 1039342 1045939432 mknod 2501 samples [usecs] 420 16984 3683480 10062548408 unlink 2500 samples [usecs] 748 89407 5174038 29544032238 mkdir 2 samples [usecs] 1002 1147 2149 2319613 rmdir 2 samples [usecs] 7819 8361 16180 131043082 rename 2500 samples [usecs] 455 47433 16164494 205366755736 getattr 3760 samples [usecs] 10 87123 405462 9069427416 parallel_rename_file 2500 samples [usecs] 455 47433 16164494 205366755736 crossdir_rename 2500 samples [usecs] 455 47433 16164494 205366755736 readpage 11 samples [usecs] 6019 52374 207441 6085743857 PASS 81d (132s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 19:33:49 (1788651229) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7655 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7655 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 19:33:55 (1788651235) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 19:33:59 (1788651239) fail_loc=0x8000060b PASS 84 (12s) == sanityn test 85: Lustre API root cache race =========== 19:34:12 (1788651252) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788651253 Finishing test test1 at 1788651258 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 19:34:22 (1788651262) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 19:34:25 (1788651265) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 19:34:28 (1788651268) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 19:34:31 (1788651271) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5847 0x16d7 0x240000400 1 5809 0x16b1 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5810 0x16b2 0x280000400 0 5848 0x16d8 0x240000400 rc1=0 and rc2=0 PASS 93 (18s) == sanityn test 94: signal vs CP callback race =========== 19:34:49 (1788651289) osc.lustre-OST0000-osc-ffff9e9fc4990000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9e9fc58c1800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9e9fc4990000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9e9fc58c1800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.240535 s, 34.1 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.94611 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 291205 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0566235 s, 72.3 kB/s osc.lustre-OST0000-osc-ffff9e9fc4990000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9e9fc58c1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e9fc4990000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e9fc58c1800.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 19:35:05 (1788651305) llite.lustre-ffff9e9fc4990000.fast_read=0 llite.lustre-ffff9e9fc58c1800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0687952 s, 30.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9e9fc4990000.fast_read=1 llite.lustre-ffff9e9fc58c1800.fast_read=1 PASS 95a (22s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 19:35:27 (1788651327) llite.lustre-ffff9e9fc4990000.fast_read=0 llite.lustre-ffff9e9fc58c1800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0184904 s, 665 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9e9fc4990000.fast_read=1 llite.lustre-ffff9e9fc58c1800.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 19:35:42 (1788651342) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 19:35:45 (1788651345) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00570959 s, 717 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 19:35:53 (1788651353) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.200201 s, 10.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 19:36:02 (1788651362) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.210259 s, 10.0 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (9s) == sanityn test 100e: DoM: read on open and file size ==== 19:36:11 (1788651371) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00663229 s, 1.8 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 19:36:19 (1788651379) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0109245 s, 375 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 19:36:26 (1788651386) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0199724 s, 205 kB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 19:36:32 (1788651392) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00520626 s, 787 kB/s PASS 101c (7s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 19:36:39 (1788651399) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 90 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf90:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (7s) == sanityn test 103: Test size correctness with lockahead ========================================================== 19:36:46 (1788651406) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788651409 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788651412 Finishing test test23 at 1788651412 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 19:37:00 (1788651420) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 19:37:03 (1788651423) fail_loc=0x416 PASS 105 (29s) == sanityn test 106a: Verify the btime via statx() ======= 19:37:32 (1788651452) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 19:37:35 (1788651455) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00942959 s, 109 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 19:37:37.000000000 -0400 Modify: 2026-09-05 19:37:37.000000000 -0400 Change: 2026-09-05 19:37:37.000000000 -0400 Birth: 2026-09-05 19:37:37.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938263 -rw-r--r-- 1 0 0 1788651457 1788651457 1788651457 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938263 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 19:37:37.000000000 -0400 Modify: 2026-09-05 19:37:37.000000000 -0400 Change: 2026-09-05 19:37:37.000000000 -0400 Birth: 2026-09-05 19:37:37.000000000 -0400 1788651457 1024 11 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 19:37:43 (1788651463) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 19:37:50 (1788651470) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.31371 s, 33.4 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5817 0x16b9 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 19:38:00 (1788651480) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.250043 s, 41.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5857 0x16e1 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7655 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 19:38:14 (1788651494) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0135298 s, 739 kB/s llite.lustre-ffff9e9fc4990000.hybrid_io=0 llite.lustre-ffff9e9fc58c1800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.70342 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 19:38:26 (1788651506) 192.168.204.111@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre (opts:) 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.111@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg411-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: kmemleak disabled oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre 192.168.204.111@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.111@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg411-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg411-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: kmemleak disabled oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre2 PASS 109 (69s) == sanityn test 110: do not grant another lock on resend ========================================================== 19:39:35 (1788651575) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 19:39:37.000000000 -0400 Modify: 2026-09-05 19:39:37.000000000 -0400 Change: 2026-09-05 19:39:37.000000000 -0400 Birth: 2026-09-05 19:39:38.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg411-client.virtnet oleg411-client.virtnet PASS 110 (175s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 19:42:30 (1788651750) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (5s) == sanityn test 112: update max-inherit in default LMV === 19:42:35 (1788651755) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 19:42:39 (1788651759) lustre-OST0000-osc-ffff9e9fd8a33000 active. lustre-OST0001-osc-ffff9e9fd8a33000 active. lustre-OST0000-osc-ffff9e9fd8a33000 active. lustre-OST0001-osc-ffff9e9fd8a33000 active. MGC192.168.204.111@tcp active. PASS 113 (8s) == sanityn test 114: implicit default LMV inherit ======== 19:42:47 (1788651767) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 19:42:51 (1788651771) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 19:42:54 (1788651774) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 19:42:57 (1788651777) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0344247 s, 595 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0892555 s, 229 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 19:43:07 (1788651787) PASS 122 (9s) == sanityn test 200: service remains healthy while able to process request ========================================================== 19:43:16 (1788651796) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1231 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1248 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.144 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1411 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2089 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.193 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2605 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0935 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1828 s, 0.1 kB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1778 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1464 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.215 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2222 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2469 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2797 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2297 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3374 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2997 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg411-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9e9fd1778000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e9fd1778000.ost_server_uuid in IDLE state after 0 sec PASS 200 (82s) cleanup: ====================================================== == sanityn test complete, duration 9772 sec ============== 19:44:39 (1788651879) === sanityn: start cleanup 19:44:41 (1788651881) === Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg411-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 19:48:52 (1788652132) ===