-----============= acceptance-small: sanityn ============----- Fri Sep 4 17:57:10 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg457-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 17:57:33 (1788559053) === Starting client oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Mount client oleg457-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg457-client.virtnet: executing check_config_client /mnt/lustre oleg457-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg457-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98f48547f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff98f4902c5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98f48547f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98f4902c5000.idle_timeout=debug disable quota as required oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 17:58:20 (1788559100) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514787 s, 20.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7646] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 17:58:23 (1788559103) PASS 0 (13s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 17:58:36 (1788559116) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (11s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 17:58:47 (1788559127) -rw-r--r-- 1 root root 0 Sep 4 17:58 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (10s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 17:58:57 (1788559137) -rw-r--r-- 1 root root 0 Sep 4 17:58 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (12s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 17:59:09 (1788559149) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 17:59:16 (1788559156) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 17:59:24 (1788559164) total 1 -rwxrwxrwx 1 root root 0 Sep 4 17:58 f2a -rwxrwxrwx 1 root root 0 Sep 4 17:58 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 4 17:58 f2a -rwxrwxrwx 1 root root 0 Sep 4 17:58 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.011939 s, 42.9 kB/s PASS 2e (11s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 17:59:34 (1788559174) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 17:59:37 (1788559177) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.649576 s, 3.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (12s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 17:59:49 (1788559189) PASS 3 (9s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 17:59:58 (1788559198) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 18:00:07 (1788559207) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 18:00:14 (1788559214) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 18:00:21 (1788559221) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 18:00:29 (1788559229) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 18:00:35 (1788559235) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 18:00:43 (1788559243) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549811 s, 19.1 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0526335 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (9s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 18:00:52 (1788559252) 1+0 records in 1+0 records out 1 byte copied, 0.00433371 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0325974 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0140836 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0101738 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0151762 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010952 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0387721 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010098 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00962444 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0231963 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0135925 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0104055 s, 0.1 kB/s PASS 10a (10s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 18:01:02 (1788559262) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0284571 s, 108 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0824924 s, 49.7 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000390797 s, 7.9 MB/s PASS 10b (9s) == sanityn test 11: execution of file opened for write should return error ============================================================== 18:01:11 (1788559271) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 18:01:19 (1788559279) warning: '-runtime' deprecated, use '-t runtime' instead - link 281 (time 1788559292.81 total 10.03 last 28.03) - link 523 (time 1788559302.82 total 20.03 last 24.19) - link 767 (time 1788559312.88 total 30.09 last 24.24) - link 1002 (time 1788559322.91 total 40.13 last 23.43) - link 1293 (time 1788559332.93 total 50.15 last 29.04) - link 1582 (time 1788559342.95 total 60.16 last 28.86) - link 1850 (time 1788559352.98 total 70.19 last 26.72) - link 2152 (time 1788559363.00 total 80.21 last 30.13) - link 2426 (time 1788559373.00 total 90.21 last 27.40) total: 2661 link in 100.01 seconds: 26.61 ops/second using seed 3040817646 running for 100 seconds - stat 3706 (time 1788559294 ; total 11 ; last 11) - stat 7378 (time 1788559305 ; total 22 ; last 11) - stat 10000 (time 1788559312 ; total 29 ; last 7) - stat 13792 (time 1788559323 ; total 40 ; last 11) - stat 18852 (time 1788559334 ; total 51 ; last 11) - stat 20000 (time 1788559339 ; total 56 ; last 5) - stat 22213 (time 1788559350 ; total 67 ; last 11) - stat 24286 (time 1788559361 ; total 78 ; last 11) - stat 26513 (time 1788559372 ; total 89 ; last 11) - stat 28843 (time 1788559383 ; total 100 ; last 11) total: 28844 stats in 100 seconds: 288.440002 stats/second - unlinked 0 (time 1788559333 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2661) error: No such file or directory total: 2660 unlinks in 81 seconds: 32.839508 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21463) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21467) - No such process PASS 12 (142s) == sanityn test 13: test directory page revocation ======= 18:03:41 (1788559421) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 18:03:51 (1788559431) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22650 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (10s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 18:04:01 (1788559441) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23231 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (10s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 18:04:11 (1788559451) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23810 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 18:04:20 (1788559460) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24392 Terminated $DIR1/$tdir/sleep 60 PASS 14c (12s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 18:04:32 (1788559472) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24977 Terminated $DIR1/$tdir/sleep 60 PASS 14d (12s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 18:04:44 (1788559484) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (61s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 18:05:45 (1788559545) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.979305 s, 8.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.46048 s, 18.2 MB/s Chance of close/open is 1 in 50 Seed set to 9552 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1788559554.528133 mapwrite 0x0875f6 thru 0x08ae96 (0x38a1 bytes) 000002[0] 1788559554.807913 punch from 0x04113e to 0x046694 (0x5557 bytes) truncating to largest ever: 0x1a16fe 000004[1] 1788559554.885719 trunc from 0x08ae97 to 0x1a16fd (0x116867 bytes) 000005[1] 1788559554.994438 punch from 0x14c759 to 0x14f449 (0x2cf1 bytes) 000007[1] 1788559555.037400 mapread 0x09ef77 thru 0x0ad696 (0xe720 bytes) 000008[1] 1788559555.172781 read 0x0ee519 thru 0x0f51fd (0x6ce5 bytes) 000010[0] 1788559555.266020 mapread 0x0aaf04 thru 0x0b45e1 (0x96de bytes) 000012[0] 1788559555.459609 mapwrite 0x702481 thru 0x707c31 (0x57b1 bytes) 000013[1] 1788559555.770507 mapwrite 0x80038b thru 0x809134 (0x8daa bytes) truncating to largest ever: 0x9bc4f8 000015[0] 1788559556.143759 trunc from 0x809135 to 0x9bc4f7 (0x1b33c3 bytes) 000016[0] 1788559556.229981 punch from 0x4d8d73 to 0x4dc6f2 (0x3980 bytes) 000018[1] 1788559556.290792 trunc from 0x9bc4f8 to 0x35d278 (0x65f27f bytes) 000019[0] 1788559556.389561 mapwrite 0x9d8ec6 thru 0x9e2545 (0x9680 bytes) 000020[1] 1788559556.601809 read 0x956577 thru 0x95ccef (0x6779 bytes) 000023[0] 1788559557.002414 mapwrite 0x77eaef thru 0x7859c2 (0x6ed4 bytes) 000024[1] 1788559557.298404 mapwrite 0x7fe0b1 thru 0x809193 (0xb0e3 bytes) 000025[0] 1788559558.018364 trunc from 0x9e2546 to 0x1cfee1 (0x812664 bytes) truncating to largest ever: 0x9cfff8 000027[0] 1788559558.285557 trunc from 0x1cfee2 to 0x9cfff7 (0x800116 bytes) 000032[0] 1788559558.500421 punch from 0x165f45 to 0x171d45 (0xbe01 bytes) 000033[0] 1788559558.566605 punch from 0x8ba558 to 0x8c73ad (0xce56 bytes) 000034[1] 1788559558.578424 mapread 0x688370 thru 0x6929ff (0xa690 bytes) 000036[1] 1788559558.785492 write 0x6eba4d thru 0x6ed985 (0x1f39 bytes) 000038[0] 1788559558.896573 mapread 0x5927d3 thru 0x59cc49 (0xa477 bytes) 000040[1] 1788559559.084688 read 0x6f07a5 thru 0x6f0892 (0xee bytes) 000041[0] 1788559559.167126 mapread 0x5a9247 thru 0x5afaf6 (0x68b0 bytes) 000042[1] 1788559559.237933 trunc from 0x9cfff8 to 0x06c4c1 (0x963b36 bytes) 000043[0] 1788559559.414798 write 0x9e3968 thru 0x9e67c0 (0x2e59 bytes) 000044[0] 1788559559.450141 trunc from 0x9e67c1 to 0x66f076 (0x37774a bytes) 000045[0] 1788559559.475398 write 0x1ca8b9 thru 0x1d68cd (0xc015 bytes) 000051[1] 1788559559.542377 mapread 0x125166 thru 0x134dd0 (0xfc6b bytes) 000053[0] 1788559559.661144 mapwrite 0x649ae6 thru 0x6597ae (0xfcc9 bytes) 000054[1] 1788559559.747919 punch from 0x16022a to 0x1633dd (0x31b4 bytes) 000056[0] 1788559559.893806 trunc from 0x66f077 to 0x165129 (0x509f4d bytes) 000059[0] 1788559560.149565 trunc from 0x16512a to 0x1ffb03 (0x9a9da bytes) 000060[1] 1788559560.246141 read 0x16f46d thru 0x172c8b (0x381f bytes) 000061[0] 1788559560.360028 trunc from 0x1ffb04 to 0x47be5b (0x27c358 bytes) 000062[1] 1788559560.415071 punch from 0x2d2fcb to 0x2d4ada (0x1b10 bytes) 000064[0] 1788559560.473060 write 0x046d08 thru 0x054e55 (0xe14e bytes) 000065[0] 1788559560.517473 punch from 0x097259 to 0x09da9a (0x6842 bytes) 000066[1] 1788559560.591247 mapread 0x234356 thru 0x242335 (0xdfe0 bytes) 000067[1] 1788559560.704429 punch from 0x46076a to 0x46d04c (0xc8e3 bytes) 000068[1] 1788559560.750194 punch from 0x09a634 to 0x0a5516 (0xaee3 bytes) 000069[0] 1788559560.797257 write 0x97daee thru 0x98cbc0 (0xf0d3 bytes) 000070[0] 1788559560.876984 punch from 0x10cf91 to 0x111ef4 (0x4f64 bytes) 000072[1] 1788559560.922907 trunc from 0x98cbc1 to 0x7d7d02 (0x1b4ebe bytes) 000076[0] 1788559561.025509 read 0x17ee4e thru 0x180b5f (0x1d12 bytes) 000077[0] 1788559561.073287 trunc from 0x7d7d03 to 0x1a5619 (0x6326e9 bytes) 000078[1] 1788559561.179485 trunc from 0x1a561a to 0x5f76d8 (0x4520bf bytes) 000079[0] 1788559561.334205 mapwrite 0x20ec08 thru 0x213d86 (0x517f bytes) 000080[1] 1788559561.660724 mapread 0x50bdf0 thru 0x519203 (0xd414 bytes) 000082[0] 1788559561.891830 punch from 0x2ba2f0 to 0x2c7805 (0xd516 bytes) 000083[1] 1788559561.934614 write 0x789a9e thru 0x78b7e7 (0x1d4a bytes) 000084[0] 1788559561.985143 write 0x982ae3 thru 0x990524 (0xda42 bytes) 000085[0] 1788559562.056129 read 0x28312c thru 0x28c986 (0x985b bytes) 000088[0] 1788559562.226136 mapwrite 0x7057c1 thru 0x7146b8 (0xeef8 bytes) 000090[0] 1788559562.526129 punch from 0x682e26 to 0x68e7e6 (0xb9c1 bytes) 000091[0] 1788559562.539575 mapread 0x92ba70 thru 0x932f19 (0x74aa bytes) 000093[1] 1788559562.671737 read 0x44ec51 thru 0x45606a (0x741a bytes) 000094[1] 1788559562.743305 mapwrite 0x3eb0cc thru 0x3ed53a (0x246f bytes) 000095[0] 1788559563.087266 trunc from 0x990525 to 0x8fdcf7 (0x9282d bytes) 000096[1] 1788559563.244206 read 0x644b30 thru 0x64947d (0x494e bytes) 000102[1] 1788559563.453723 read 0x2e4f0d thru 0x2f05cc (0xb6c0 bytes) 000104[0] 1788559563.495548 trunc from 0x8fdcf8 to 0x859c15 (0xa40e2 bytes) 000105[0] 1788559563.617048 punch from 0x450558 to 0x45ec91 (0xe73a bytes) 000106[1] 1788559563.664573 punch from 0x357ceb to 0x3614a9 (0x97bf bytes) 000108[1] 1788559563.807412 mapwrite 0x72347e thru 0x723aca (0x64d bytes) 000110[0] 1788559564.013047 punch from 0x83ad7c to 0x8425a8 (0x782d bytes) 000111[1] 1788559564.066632 mapread 0x37b578 thru 0x382c68 (0x76f1 bytes) 000112[0] 1788559564.130802 trunc from 0x859c16 to 0x8c0564 (0x6694f bytes) 000113[0] 1788559564.211086 read 0x107868 thru 0x10857b (0xd14 bytes) 000114[0] 1788559564.274750 mapread 0x8aff5e thru 0x8b5600 (0x56a3 bytes) 000116[1] 1788559564.325376 write 0x316016 thru 0x31e704 (0x86ef bytes) 000119[1] 1788559564.419413 punch from 0x44859b to 0x4501e0 (0x7c46 bytes) 000120[0] 1788559564.456956 trunc from 0x8c0565 to 0x567876 (0x358cee bytes) 000124[1] 1788559564.682521 mapread 0x488275 thru 0x489c74 (0x1a00 bytes) 000127[0] 1788559564.778033 trunc from 0x567877 to 0x666d67 (0xff4f1 bytes) 000128[0] 1788559564.836262 trunc from 0x666d68 to 0x0fb06e (0x56bcf9 bytes) 000129[1] 1788559564.879242 mapwrite 0x3b06b3 thru 0x3b86cf (0x801d bytes) 000131[0] 1788559565.122631 read 0x0def93 thru 0x0e9ec4 (0xaf32 bytes) 000132[0] 1788559565.257109 write 0x7c0797 thru 0x7cfa43 (0xf2ad bytes) 000134[0] 1788559565.302609 mapread 0x102784 thru 0x10e8f5 (0xc172 bytes) 000137[1] 1788559565.352013 write 0x035540 thru 0x036a0b (0x14cc bytes) 000139[1] 1788559565.475790 trunc from 0x7cfa44 to 0x4fb7a2 (0x2d42a1 bytes) 000140[1] 1788559565.578820 mapwrite 0x13ff5e thru 0x14f0df (0xf182 bytes) 000142[1] 1788559565.815739 mapread 0x2f5416 thru 0x2f6ff3 (0x1bde bytes) 000144[0] 1788559565.881237 read 0x450415 thru 0x453e9c (0x3a88 bytes) 000145[1] 1788559565.941085 trunc from 0x4fb7a3 to 0x881a05 (0x386263 bytes) 000146[1] 1788559566.056004 mapread 0x0f8224 thru 0x10341a (0xb1f7 bytes) 000147[1] 1788559566.068994 read 0x6878ab thru 0x68fb41 (0x8297 bytes) 000149[0] 1788559566.113072 trunc from 0x881a06 to 0x3a5fe0 (0x4dba25 bytes) 000152[1] 1788559566.196137 punch from 0x249753 to 0x253b78 (0xa426 bytes) 000153[1] 1788559566.235949 mapwrite 0x9141b0 thru 0x923d59 (0xfbaa bytes) 000154[0] 1788559566.437470 write 0x5c82c6 thru 0x5d227a (0x9fb5 bytes) 000155[0] 1788559566.491532 write 0x5f04b9 thru 0x5fae31 (0xa979 bytes) 000157[0] 1788559566.561806 mapwrite 0x936289 thru 0x9408e6 (0xa65e bytes) 000158[0] 1788559566.791877 read 0x4e176d thru 0x4ea896 (0x912a bytes) 000159[1] 1788559566.822747 trunc from 0x9408e7 to 0x44761f (0x4f92c7 bytes) 000161[0] 1788559566.954312 mapwrite 0x30c75c thru 0x31434b (0x7bf0 bytes) 000163[1] 1788559567.100435 mapread 0x280439 thru 0x287623 (0x71eb bytes) 000164[1] 1788559567.172620 trunc from 0x447620 to 0x7089a9 (0x2c138a bytes) 000166[0] 1788559567.225739 punch from 0x399e5b to 0x3a9314 (0xf4ba bytes) 000168[1] 1788559567.275876 trunc from 0x7089aa to 0x890218 (0x18786f bytes) 000169[1] 1788559567.319725 read 0x39de55 thru 0x3aa2a2 (0xc44e bytes) 000170[0] 1788559567.329566 mapread 0x44f049 thru 0x45c812 (0xd7ca bytes) 000171[1] 1788559567.403013 mapread 0x225d5f thru 0x227052 (0x12f4 bytes) 000174[1] 1788559567.448499 punch from 0x4cdd5d to 0x4d9037 (0xb2db bytes) 000175[0] 1788559567.468905 trunc from 0x890219 to 0x29ead0 (0x5f1748 bytes) 000176[0] 1788559567.506271 read 0x2030a8 thru 0x207c67 (0x4bc0 bytes) 000178[1] 1788559567.646592 write 0x253ae4 thru 0x25c676 (0x8b93 bytes) 000182[0] 1788559567.739686 mapwrite 0x358fd9 thru 0x35fbbe (0x6be6 bytes) 000183[1] 1788559567.881128 punch from 0x180227 to 0x181277 (0x1051 bytes) 000185[1] 1788559567.924424 read 0x025baf thru 0x033f55 (0xe3a7 bytes) 000187[1] 1788559568.060645 mapwrite 0x9c2603 thru 0x9cae25 (0x8823 bytes) 000188[1] 1788559568.354045 punch from 0x4d1937 to 0x4e08ab (0xef75 bytes) 000189[0] 1788559568.367562 punch from 0x78da2b to 0x79c0fa (0xe6d0 bytes) 000190[0] 1788559568.423769 mapwrite 0x513d29 thru 0x51a54b (0x6823 bytes) 000192[1] 1788559568.583254 mapread 0x1bd527 thru 0x1c062b (0x3105 bytes) 000193[0] 1788559568.603848 trunc from 0x9cae26 to 0x2080e9 (0x7c2d3c bytes) 000194[0] 1788559568.653678 mapwrite 0x62b29c thru 0x62c087 (0xdec bytes) 000195[1] 1788559568.811139 trunc from 0x62c088 to 0x0b2146 (0x579f41 bytes) 000197[1] 1788559568.860877 mapread 0x030d27 thru 0x037e3d (0x7117 bytes) 000200[0] 1788559568.986510 punch from 0x04700f to 0x054bd5 (0xdbc7 bytes) 000201[1] 1788559569.028636 punch from 0x05d89f to 0x062c28 (0x538a bytes) 000202[1] 1788559569.068067 punch from 0x01ad70 to 0x027308 (0xc599 bytes) 000203[1] 1788559569.073303 punch from 0x021bd4 to 0x027d7b (0x61a8 bytes) 000204[1] 1788559569.078040 trunc from 0x0b2147 to 0x253481 (0x1a133b bytes) 000206[1] 1788559569.133656 punch from 0x0cd3b4 to 0x0d3ca5 (0x68f2 bytes) 000207[1] 1788559569.140608 read 0x18b1e9 thru 0x197d38 (0xcb50 bytes) 000208[1] 1788559569.221222 mapwrite 0x76b242 thru 0x770df3 (0x5bb2 bytes) 000209[1] 1788559569.445990 mapwrite 0x34593a thru 0x346b0e (0x11d5 bytes) 000212[0] 1788559569.708458 punch from 0x032d93 to 0x03badf (0x8d4d bytes) 000213[1] 1788559569.807288 mapwrite 0x0744e5 thru 0x07f789 (0xb2a5 bytes) 000214[1] 1788559570.165968 punch from 0x045834 to 0x04bb84 (0x6351 bytes) 000216[1] 1788559570.183258 punch from 0x3c5023 to 0x3d0d29 (0xbd07 bytes) 000217[1] 1788559570.202300 trunc from 0x770df4 to 0x4d69a2 (0x29a451 bytes) 000218[1] 1788559570.285052 mapwrite 0x4f2fc4 thru 0x4fa48f (0x74cc bytes) 000219[0] 1788559570.661972 write 0x91713b thru 0x9216f6 (0xa5bc bytes) 000220[0] 1788559570.747314 punch from 0x12c317 to 0x134a1e (0x8708 bytes) 000221[1] 1788559570.797362 mapwrite 0x282202 thru 0x28a00d (0x7e0c bytes) 000222[1] 1788559571.379942 mapwrite 0x8b99cd thru 0x8c6ea3 (0xd4d7 bytes) 000224[0] 1788559571.692778 mapread 0x2d5744 thru 0x2e230a (0xcbc7 bytes) 000229[1] 1788559571.863508 read 0x1a9b54 thru 0x1ac928 (0x2dd5 bytes) 000231[0] 1788559571.900296 mapread 0x71dd83 thru 0x721870 (0x3aee bytes) 000233[0] 1788559571.997323 write 0x0d705a thru 0x0e67b3 (0xf75a bytes) 000234[0] 1788559572.044717 read 0x2b8f7e thru 0x2bbbc2 (0x2c45 bytes) 000235[1] 1788559572.061925 trunc from 0x9216f7 to 0x585c18 (0x39bade bytes) 000236[0] 1788559572.129367 mapwrite 0x799bd0 thru 0x79f534 (0x5965 bytes) 000237[0] 1788559572.299055 trunc from 0x79f535 to 0x9cd0f1 (0x22dbbd bytes) 000239[1] 1788559572.346869 read 0x5e2356 thru 0x5f04bf (0xe16a bytes) 000240[1] 1788559572.377635 mapwrite 0x9baa00 thru 0x9c17b0 (0x6db1 bytes) 000241[0] 1788559572.471793 punch from 0x06f4ab to 0x071817 (0x236d bytes) 000242[0] 1788559572.503488 mapread 0x2c79a4 thru 0x2c9e3b (0x2498 bytes) 000243[0] 1788559572.540450 trunc from 0x9cd0f2 to 0x693810 (0x3398e1 bytes) 000245[0] 1788559572.586741 write 0x652fbb thru 0x6572e9 (0x432f bytes) 000246[0] 1788559572.601591 write 0x0874f3 thru 0x08bbcc (0x46da bytes) 000248[0] 1788559572.644733 read 0x60e9bb thru 0x60ff8c (0x15d2 bytes) 000249[0] 1788559572.671407 mapread 0x2759f2 thru 0x27799c (0x1fab bytes) 000250[1] 1788559572.722461 mapread 0x5c6fc1 thru 0x5ca653 (0x3693 bytes) 000254[0] 1788559572.973467 read 0x0b57b9 thru 0x0c4141 (0xe989 bytes) 000255[1] 1788559572.993391 mapwrite 0x07d1ca thru 0x087dcc (0xac03 bytes) 000262[1] 1788559573.214677 read 0x27bb3d thru 0x288d16 (0xd1da bytes) 000265[0] 1788559573.335670 write 0x0c21a0 thru 0x0c232b (0x18c bytes) 000268[1] 1788559573.391836 read 0x5554e4 thru 0x55e12b (0x8c48 bytes) 000269[0] 1788559573.431012 mapread 0x389423 thru 0x38b855 (0x2433 bytes) 000270[0] 1788559573.469770 trunc from 0x693811 to 0x8c1031 (0x22d821 bytes) 000273[0] 1788559573.566102 read 0x686e07 thru 0x691ed2 (0xb0cc bytes) 000274[1] 1788559573.624989 read 0x6d0451 thru 0x6dd643 (0xd1f3 bytes) 000275[0] 1788559573.677991 trunc from 0x8c1032 to 0x423252 (0x49dddf bytes) 000276[0] 1788559573.771134 trunc from 0x423253 to 0x6718a9 (0x24e657 bytes) 000277[1] 1788559573.813072 mapwrite 0x530150 thru 0x537642 (0x74f3 bytes) 000278[0] 1788559574.050410 punch from 0x3341a3 to 0x337f53 (0x3db1 bytes) 000280[0] 1788559574.105643 punch from 0x237427 to 0x24349c (0xc076 bytes) 000282[0] 1788559574.157362 mapwrite 0x17aa10 thru 0x18386b (0x8e5c bytes) 000284[0] 1788559574.353687 trunc from 0x6718aa to 0x2b03a7 (0x3c1502 bytes) 000285[1] 1788559574.440584 read 0x09930c thru 0x09fbd0 (0x68c5 bytes) 000286[0] 1788559574.533503 punch from 0x067728 to 0x06a965 (0x323e bytes) 000288[1] 1788559574.586417 read 0x14853b thru 0x14ac65 (0x272b bytes) 000290[1] 1788559574.701144 mapwrite 0x132598 thru 0x14225d (0xfcc6 bytes) 000291[0] 1788559575.019984 mapread 0x278bc4 thru 0x2871f6 (0xe633 bytes) 000293[0] 1788559575.071609 write 0x282c84 thru 0x28aca7 (0x8024 bytes) 000294[0] 1788559575.098013 mapread 0x23a36d thru 0x23b5a5 (0x1239 bytes) 000295[1] 1788559575.126935 read 0x224369 thru 0x2331a3 (0xee3b bytes) 000296[1] 1788559575.209910 mapread 0x1335f8 thru 0x1377ad (0x41b6 bytes) 000298[0] 1788559575.225085 read 0x147d2c thru 0x14c73a (0x4a0f bytes) 000301[1] 1788559575.338974 punch from 0x2ad16f to 0x2ae108 (0xf9a bytes) 000302[1] 1788559575.442465 read 0x07c01b thru 0x085087 (0x906d bytes) 000304[0] 1788559575.543882 mapwrite 0x36db04 thru 0x36eaa5 (0xfa2 bytes) 000307[0] 1788559575.841513 punch from 0x2bd4e6 to 0x2cb309 (0xde24 bytes) 000308[0] 1788559575.901280 punch from 0x137e8a to 0x143524 (0xb69b bytes) 000309[1] 1788559575.969880 trunc from 0x36eaa6 to 0x042360 (0x32c745 bytes) 000311[1] 1788559576.060262 trunc from 0x042361 to 0x6e9940 (0x6a75e0 bytes) 000312[0] 1788559576.086884 write 0x418bb5 thru 0x41d428 (0x4874 bytes) 000313[0] 1788559576.130063 mapwrite 0x356b04 thru 0x35b07d (0x457a bytes) 000315[0] 1788559576.326596 read 0x3f441d thru 0x3f86b8 (0x429c bytes) 000316[0] 1788559576.358753 mapread 0x627b2e thru 0x63605b (0xe52e bytes) 000317[0] 1788559576.408809 mapwrite 0x140681 thru 0x142490 (0x1e10 bytes) 000318[1] 1788559576.551664 mapwrite 0x9d2c64 thru 0x9d3a55 (0xdf2 bytes) 000319[0] 1788559576.984197 punch from 0x07da3c to 0x08a6db (0xcca0 bytes) 000320[1] 1788559577.024136 mapwrite 0x23bc31 thru 0x23f9d5 (0x3da5 bytes) 000321[0] 1788559577.127100 read 0x8b8538 thru 0x8c1d0d (0x97d6 bytes) 000324[1] 1788559577.191238 mapwrite 0x4524df thru 0x45a4a6 (0x7fc8 bytes) 000326[1] 1788559577.281584 write 0x9b66ec thru 0x9bc67a (0x5f8f bytes) 000327[0] 1788559577.348774 punch from 0x824fe9 to 0x82aff3 (0x600b bytes) 000328[0] 1788559577.411527 read 0x600c54 thru 0x606d39 (0x60e6 bytes) 000329[1] 1788559577.461778 punch from 0x0358d1 to 0x03fa8e (0xa1be bytes) 000330[1] 1788559577.552872 read 0x6d5470 thru 0x6e526c (0xfdfd bytes) 000331[0] 1788559577.611685 write 0x348be7 thru 0x34fe00 (0x721a bytes) 000337[0] 1788559577.812446 write 0x3cb40a thru 0x3d72c8 (0xbebf bytes) 000338[0] 1788559577.834122 write 0x32506b thru 0x32c9fd (0x7993 bytes) 000339[0] 1788559577.896019 trunc from 0x9d3a56 to 0x022483 (0x9b15d2 bytes) 000340[0] 1788559577.963231 mapwrite 0x8a5341 thru 0x8a5802 (0x4c2 bytes) 000343[0] 1788559578.160837 mapread 0x2b948e thru 0x2bcfec (0x3b5f bytes) 000345[1] 1788559578.209041 mapwrite 0x35fc0c thru 0x36659b (0x6990 bytes) 000347[0] 1788559578.406804 write 0x00128f thru 0x00272e (0x14a0 bytes) 000349[0] 1788559578.493890 read 0x0f0935 thru 0x0f50a6 (0x4772 bytes) 000350[0] 1788559578.544847 trunc from 0x8a5803 to 0x131e80 (0x773982 bytes) 000351[0] 1788559578.615102 write 0x0d11ca thru 0x0d3e80 (0x2cb7 bytes) 000355[0] 1788559578.729871 trunc from 0x131e81 to 0x4c4f22 (0x3930a2 bytes) 000356[0] 1788559578.758473 punch from 0x3db192 to 0x3dd4f6 (0x2365 bytes) 000358[0] 1788559578.801096 punch from 0x33e530 to 0x3452bb (0x6d8c bytes) 000362[0] 1788559578.879601 mapwrite 0x92a83e thru 0x935002 (0xa7c5 bytes) 000364[0] 1788559579.139595 mapwrite 0x515a6e thru 0x516f99 (0x152c bytes) 000365[1] 1788559579.324679 mapread 0x8b40ac thru 0x8b71d7 (0x312c bytes) 000366[0] 1788559579.407905 mapwrite 0x9e5d3d thru 0x9f56d9 (0xf99d bytes) 000369[1] 1788559579.586943 punch from 0x238b89 to 0x2482bb (0xf733 bytes) 000370[1] 1788559579.639482 mapread 0x274041 thru 0x278095 (0x4055 bytes) 000371[0] 1788559579.686493 punch from 0x56cbb0 to 0x57a66c (0xdabd bytes) 000372[1] 1788559579.707247 read 0x8f98a9 thru 0x905d1a (0xc472 bytes) 000375[1] 1788559579.780110 punch from 0x700731 to 0x701d81 (0x1651 bytes) 000377[1] 1788559579.803619 trunc from 0x9f56da to 0x812ff3 (0x1e26e6 bytes) 000380[0] 1788559579.894785 mapread 0x6c016d thru 0x6ccf8c (0xce20 bytes) 000381[1] 1788559579.970942 mapread 0x404a8b thru 0x41319d (0xe713 bytes) 000382[0] 1788559580.035424 punch from 0x7a5618 to 0x7a6329 (0xd12 bytes) 000384[0] 1788559580.120677 mapread 0x0ce538 thru 0x0d6d02 (0x87cb bytes) 000386[1] 1788559580.227538 mapread 0x1f34ce thru 0x1f93b8 (0x5eeb bytes) 000387[1] 1788559580.265328 mapwrite 0x295a51 thru 0x296ee7 (0x1497 bytes) 000388[0] 1788559580.434110 write 0x151e69 thru 0x1555c4 (0x375c bytes) 000389[1] 1788559580.477978 mapwrite 0x309db4 thru 0x315d9f (0xbfec bytes) 000390[0] 1788559580.668001 punch from 0x4c41c2 to 0x4c4fa0 (0xddf bytes) 000391[1] 1788559580.690278 punch from 0x08d673 to 0x09613c (0x8aca bytes) 000393[1] 1788559580.729611 mapread 0x5ba134 thru 0x5c0207 (0x60d4 bytes) 000395[0] 1788559580.774977 read 0x7cdc61 thru 0x7d4576 (0x6916 bytes) 000396[0] 1788559580.822668 trunc from 0x812ff4 to 0x3e60ff (0x42cef4 bytes) 000397[1] 1788559580.978239 mapread 0x28150a thru 0x286247 (0x4d3e bytes) 000398[0] 1788559581.093974 write 0x9b467c thru 0x9bf3c0 (0xad45 bytes) 000399[0] 1788559581.119869 trunc from 0x9bf3c1 to 0x85672a (0x168c96 bytes) 000403[0] 1788559581.223996 trunc from 0x85672b to 0x014cc3 (0x841a67 bytes) 000404[0] 1788559581.268295 mapwrite 0x9a7e8c thru 0x9aab67 (0x2cdc bytes) 000408[1] 1788559581.513349 mapwrite 0x300075 thru 0x3082a7 (0x8233 bytes) 000411[1] 1788559581.706416 write 0x124de6 thru 0x129058 (0x4273 bytes) 000413[1] 1788559581.770989 mapread 0x60ae6d thru 0x6176d7 (0xc86b bytes) 000414[0] 1788559581.851599 read 0x4eb01f thru 0x4f8ae8 (0xdaca bytes) 000415[0] 1788559581.871834 mapwrite 0x108fbb thru 0x1129ec (0x9a32 bytes) 000418[1] 1788559582.141943 read 0x4ee696 thru 0x4f2243 (0x3bae bytes) 000421[1] 1788559582.176074 read 0x2776fa thru 0x277dc6 (0x6cd bytes) 000422[0] 1788559582.200883 trunc from 0x9aab68 to 0x20a79b (0x7a03cc bytes) 000423[0] 1788559582.273305 read 0x00febe thru 0x012bae (0x2cf1 bytes) 000426[0] 1788559582.394818 read 0x1faddf thru 0x200b48 (0x5d6a bytes) 000427[0] 1788559582.395562 read 0x177311 thru 0x17e8b5 (0x75a5 bytes) 000432[0] 1788559582.479693 punch from 0x167ed4 to 0x176475 (0xe5a2 bytes) 000435[0] 1788559582.558860 trunc from 0x20a79c to 0x1b66bc (0x540df bytes) 000438[1] 1788559582.606019 mapwrite 0x3fed5b thru 0x40839e (0x9644 bytes) 000440[1] 1788559582.893481 mapread 0x1ac16b thru 0x1ae255 (0x20eb bytes) 000441[1] 1788559582.958370 mapread 0x1e367f thru 0x1e38b6 (0x238 bytes) 000442[0] 1788559582.976568 mapread 0x0fcc3e thru 0x105e63 (0x9226 bytes) 000444[1] 1788559583.218232 read 0x097587 thru 0x09f8b8 (0x8332 bytes) 000447[0] 1788559583.238033 mapwrite 0x165c89 thru 0x1702d9 (0xa651 bytes) 000448[1] 1788559583.499450 trunc from 0x40839f to 0x98d366 (0x584fc8 bytes) 000450[1] 1788559583.598114 read 0x47f1b3 thru 0x4809e9 (0x1837 bytes) 000451[0] 1788559583.609916 write 0x2ab43f thru 0x2ae5ba (0x317c bytes) 000453[0] 1788559583.663526 trunc from 0x98d367 to 0x8c9290 (0xc40d6 bytes) 000454[1] 1788559583.698473 mapread 0x86699b thru 0x872c37 (0xc29d bytes) 000455[1] 1788559583.757799 write 0x229593 thru 0x22ebd2 (0x5640 bytes) 000458[1] 1788559583.834988 mapwrite 0x5351da thru 0x538f29 (0x3d50 bytes) 000460[0] 1788559583.957555 trunc from 0x8c9291 to 0x61714f (0x2b2141 bytes) 000461[1] 1788559584.076858 read 0x08c4bd thru 0x08d602 (0x1146 bytes) 000462[1] 1788559584.156386 mapwrite 0x2baf1e thru 0x2bed40 (0x3e23 bytes) 000463[1] 1788559584.317512 write 0x65bc9f thru 0x6671cf (0xb531 bytes) 000465[1] 1788559584.372737 read 0x26aefb thru 0x278fd5 (0xe0db bytes) 000466[0] 1788559584.383448 read 0x621b43 thru 0x622267 (0x725 bytes) 000467[1] 1788559584.458414 write 0x3ee6ad thru 0x3f0150 (0x1aa4 bytes) 000468[0] 1788559584.517518 mapread 0x2f9003 thru 0x2ffd65 (0x6d63 bytes) 000472[1] 1788559584.629694 read 0x212bc2 thru 0x214459 (0x1898 bytes) 000473[0] 1788559584.644903 write 0x164b42 thru 0x169b18 (0x4fd7 bytes) 000475[0] 1788559584.695939 mapread 0x16af40 thru 0x17a6e5 (0xf7a6 bytes) 000479[1] 1788559584.793444 punch from 0x26d299 to 0x27c066 (0xedce bytes) 000480[0] 1788559584.861081 mapwrite 0x189a00 thru 0x18fbec (0x61ed bytes) 000481[1] 1788559585.175467 read 0x015ba7 thru 0x01c1a5 (0x65ff bytes) 000483[0] 1788559585.234743 mapwrite 0x5df627 thru 0x5ee3d0 (0xedaa bytes) 000484[0] 1788559585.418470 mapread 0x5d457d thru 0x5dde30 (0x98b4 bytes) 000485[1] 1788559585.459356 punch from 0x23b676 to 0x2484d4 (0xce5f bytes) 000486[1] 1788559585.504138 trunc from 0x6671d0 to 0x1a6f9c (0x4c0233 bytes) 000487[1] 1788559585.556223 punch from 0x04a082 to 0x04d1b1 (0x3130 bytes) 000488[1] 1788559585.584586 trunc from 0x1a6f9d to 0x5d7d2e (0x430d92 bytes) 000489[0] 1788559585.630450 write 0x42d36f thru 0x43a70f (0xd3a1 bytes) 000491[1] 1788559585.689481 write 0x0c470c thru 0x0d3047 (0xe93c bytes) 000492[0] 1788559585.723342 mapwrite 0x334a66 thru 0x33669d (0x1c38 bytes) 000493[1] 1788559585.907625 mapread 0x35d9aa thru 0x36a71c (0xcd73 bytes) 000494[1] 1788559585.976371 punch from 0x19c2a5 to 0x1aa34e (0xe0aa bytes) 000495[1] 1788559586.027231 mapread 0x412358 thru 0x4217ae (0xf457 bytes) 000496[1] 1788559586.121273 mapread 0x522290 thru 0x522888 (0x5f9 bytes) 000497[1] 1788559586.127770 mapwrite 0x182743 thru 0x1891f4 (0x6ab2 bytes) 000499[1] 1788559586.263418 mapwrite 0x7815d8 thru 0x78e3e6 (0xce0f bytes) 000500[1] 1788559586.495606 read 0x0609f1 thru 0x06fa1f (0xf02f bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9586 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole 000005[1] 1788559587.632503 write_OD 0x9cd000 thru 0x9cffff (0x3000 bytes) truncating to largest ever: 0x1f5b62 000006[1] 1788559587.726224 trunc from 0x9d0000 to 0x1f5b61 (0x7da49e bytes) truncating to largest ever: 0x586300 000010[0] 1788559587.848613 trunc from 0x1f5b62 to 0x5862ff (0x39079e bytes) 000012[1] 1788559587.942835 punch from 0x347f97 to 0x353a98 (0xbb02 bytes) truncating to largest ever: 0x7d24e0 000013[1] 1788559587.988537 trunc from 0x586300 to 0x7d24df (0x24c1e0 bytes) 000016[0] 1788559588.020779 mapwrite 0x940000 thru 0x944c93 (0x4c94 bytes) 000017[1] 1788559588.278218 punch from 0x3ed9ee to 0x3ee4bb (0xace bytes) 000018[1] 1788559588.348384 mapwrite 0x368000 thru 0x36aec3 (0x2ec4 bytes) 000019[1] 1788559588.438177 write_OD 0x72f000 thru 0x738fff (0xa000 bytes) 000023[1] 1788559588.499757 punch from 0x24eee8 to 0x25c236 (0xd34f bytes) 000024[1] 1788559588.519260 punch from 0x5603ac to 0x565f19 (0x5b6e bytes) 000026[0] 1788559588.549616 mapwrite 0x2ae000 thru 0x2b4d84 (0x6d85 bytes) truncating to largest ever: 0x9c96b1 000027[0] 1788559588.734015 trunc from 0x944c94 to 0x9c96b0 (0x84a1d bytes) 000028[1] 1788559588.782383 write_OD 0x481000 thru 0x48afff (0xa000 bytes) 000031[1] 1788559588.885731 mapread 0x72e000 thru 0x7340b0 (0x60b1 bytes) 000032[1] 1788559588.910416 mapwrite 0x73d000 thru 0x73f07a (0x207b bytes) 000035[0] 1788559589.081330 read 0x453000 thru 0x4620b5 (0xf0b6 bytes) 000037[1] 1788559589.128602 trunc from 0x9c96b1 to 0x31dac4 (0x6abbec bytes) 000038[1] 1788559589.166573 trunc from 0x31dac5 to 0x64d53d (0x32fa79 bytes) 000041[0] 1788559589.239981 mapwrite 0x356000 thru 0x35cda7 (0x6da8 bytes) 000042[0] 1788559589.373256 read 0x1ad000 thru 0x1b27ec (0x57ed bytes) 000045[1] 1788559589.425478 trunc from 0x64d53e to 0x93f855 (0x2f2318 bytes) 000046[0] 1788559589.466400 trunc from 0x93f856 to 0x186380 (0x7b94d5 bytes) 000047[1] 1788559589.526502 punch from 0x1755f5 to 0x1761c7 (0xbd3 bytes) 000048[1] 1788559589.560715 trunc from 0x186381 to 0x53fbcc (0x3b984c bytes) 000049[1] 1788559589.617282 trunc from 0x53fbcd to 0x4baacd (0x850ff bytes) 000050[0] 1788559589.686015 mapread 0x05f000 thru 0x062e9f (0x3ea0 bytes) 000054[1] 1788559589.812120 mapwrite 0x785000 thru 0x786df2 (0x1df3 bytes) 000055[1] 1788559589.906883 punch from 0x39ffc7 to 0x3ac18a (0xc1c4 bytes) 000056[1] 1788559589.911372 mapread 0x747000 thru 0x74eecf (0x7ed0 bytes) 000058[0] 1788559589.921061 read 0x362000 thru 0x369c18 (0x7c19 bytes) 000059[0] 1788559589.943832 mapwrite 0x539000 thru 0x543ac4 (0xaac5 bytes) 000060[1] 1788559590.021034 punch from 0x22e1c8 to 0x2391de (0xb017 bytes) 000062[1] 1788559590.054215 read_OD 0x751000 thru 0x75afff (0xa000 bytes) 000063[0] 1788559590.083845 trunc from 0x786df3 to 0x2a2409 (0x4e49e9 bytes) 000064[0] 1788559590.156998 punch from 0x15fce1 to 0x165003 (0x5323 bytes) 000065[1] 1788559590.167316 write_OD 0x56d000 thru 0x56efff (0x2000 bytes) 000066[1] 1788559590.223568 write_OD 0x42c000 thru 0x42ffff (0x4000 bytes) 000068[1] 1788559590.276605 punch from 0x103ed1 to 0x10cd1c (0x8e4c bytes) 000071[1] 1788559590.313040 write_OD 0x503000 thru 0x50dfff (0xb000 bytes) 000072[0] 1788559590.385187 trunc from 0x56f000 to 0x5b0473 (0x41474 bytes) 000073[1] 1788559590.418899 mapwrite 0x8d1000 thru 0x8ddf41 (0xcf42 bytes) 000074[1] 1788559590.586610 write_OD 0x3aa000 thru 0x3b8fff (0xf000 bytes) 000077[1] 1788559590.714985 trunc from 0x8ddf42 to 0x738ec2 (0x1a507f bytes) 000079[1] 1788559590.804799 punch from 0x031769 to 0x03f909 (0xe1a1 bytes) 000080[0] 1788559590.832511 punch from 0x3d3ba4 to 0x3d62f3 (0x2750 bytes) 000081[1] 1788559590.876611 write_OD 0x80f000 thru 0x81cfff (0xe000 bytes) 000083[0] 1788559590.944312 read 0x364000 thru 0x36bfa2 (0x7fa3 bytes) 000087[0] 1788559590.957774 punch from 0x68caa0 to 0x695d1b (0x927c bytes) 000088[0] 1788559591.022873 mapwrite 0x58c000 thru 0x59077d (0x477e bytes) 000090[1] 1788559591.377126 read_OD 0x4c3000 thru 0x4cefff (0xc000 bytes) 000091[1] 1788559591.428409 trunc from 0x81d000 to 0x5623ec (0x2bac13 bytes) 000092[1] 1788559591.480245 read_OD 0x04b000 thru 0x04cfff (0x2000 bytes) 000093[0] 1788559591.495425 trunc from 0x5623ed to 0x5ea929 (0x8853d bytes) 000094[1] 1788559591.539567 trunc from 0x5ea92a to 0x09df47 (0x54c9e2 bytes) 000096[0] 1788559591.585366 punch from 0x08faae to 0x09b7d4 (0xbd27 bytes) 000098[1] 1788559591.600891 write_OD 0x855000 thru 0x861fff (0xd000 bytes) 000102[1] 1788559591.680915 trunc from 0x862000 to 0x14ed39 (0x7132c6 bytes) 000103[1] 1788559591.732052 mapwrite 0x601000 thru 0x6080cb (0x70cc bytes) 000104[0] 1788559591.826720 write 0x947000 thru 0x9499c6 (0x29c7 bytes) 000105[0] 1788559591.850396 punch from 0x0d49b9 to 0x0de50b (0x9b53 bytes) 000106[1] 1788559591.863507 trunc from 0x9499c7 to 0x164155 (0x7e5871 bytes) 000107[1] 1788559591.937412 trunc from 0x164156 to 0x9afb53 (0x84b9fe bytes) 000108[0] 1788559591.973677 mapwrite 0x093000 thru 0x0a07ef (0xd7f0 bytes) 000110[0] 1788559592.124129 punch from 0x812f80 to 0x814d13 (0x1d94 bytes) 000112[0] 1788559592.159821 write 0x4d3000 thru 0x4d4d09 (0x1d0a bytes) 000115[1] 1788559592.201988 write_OD 0x02a000 thru 0x030fff (0x7000 bytes) 000116[0] 1788559592.263061 mapread 0x1cb000 thru 0x1d4abe (0x9abf bytes) 000117[0] 1788559592.293878 mapread 0x39f000 thru 0x3ae4d5 (0xf4d6 bytes) 000120[0] 1788559592.338750 punch from 0x2af459 to 0x2b304e (0x3bf6 bytes) 000121[1] 1788559592.369013 trunc from 0x9afb54 to 0x685695 (0x32a4be bytes) 000123[0] 1788559592.451417 mapwrite 0x151000 thru 0x1579b7 (0x69b8 bytes) 000124[0] 1788559592.537336 read 0x093000 thru 0x09cb05 (0x9b06 bytes) 000126[1] 1788559592.560871 mapread 0x304000 thru 0x3119db (0xd9dc bytes) 000127[1] 1788559592.608481 write_OD 0x05a000 thru 0x068fff (0xf000 bytes) 000129[0] 1788559592.654981 write 0x2f9000 thru 0x302e55 (0x9e56 bytes) 000130[1] 1788559592.702382 write_OD 0x012000 thru 0x01efff (0xd000 bytes) 000133[0] 1788559592.849095 punch from 0x3a9282 to 0x3b6447 (0xd1c6 bytes) 000134[1] 1788559592.915313 write_OD 0x3a6000 thru 0x3a7fff (0x2000 bytes) 000136[1] 1788559593.018259 punch from 0x3f83c3 to 0x3f9479 (0x10b7 bytes) 000137[0] 1788559593.066216 punch from 0x66c40b to 0x672bd0 (0x67c6 bytes) 000138[1] 1788559593.091001 trunc from 0x685696 to 0x5242a0 (0x1613f5 bytes) 000139[0] 1788559593.127284 read 0x467000 thru 0x46d2b3 (0x62b4 bytes) 000144[1] 1788559593.158117 trunc from 0x5242a1 to 0x0bd3ed (0x466eb3 bytes) 000146[1] 1788559593.237839 mapread 0x081000 thru 0x083c40 (0x2c41 bytes) 000148[1] 1788559593.285263 mapwrite 0x4e1000 thru 0x4f03c2 (0xf3c3 bytes) 000149[0] 1788559593.404478 punch from 0x476b07 to 0x481273 (0xa76d bytes) 000151[1] 1788559593.478780 read_OD 0x141000 thru 0x145fff (0x5000 bytes) 000153[1] 1788559593.542707 trunc from 0x4f03c3 to 0x73c4c0 (0x24c0fe bytes) 000154[0] 1788559593.621612 mapwrite 0x170000 thru 0x17e338 (0xe339 bytes) 000155[0] 1788559593.838378 trunc from 0x73c4c1 to 0x06c8da (0x6cfbe6 bytes) 000158[1] 1788559593.909761 read_OD 0x029000 thru 0x02dfff (0x5000 bytes) 000159[0] 1788559593.950807 punch from 0x04bc8f to 0x04e060 (0x23d2 bytes) 000161[0] 1788559593.991382 read 0x035000 thru 0x038796 (0x3797 bytes) 000162[1] 1788559594.048622 mapread 0x017000 thru 0x01af91 (0x3f92 bytes) 000164[1] 1788559594.125457 mapwrite 0x324000 thru 0x32caa0 (0x8aa1 bytes) 000166[1] 1788559594.485092 mapwrite 0x49a000 thru 0x4a8cdf (0xece0 bytes) 000167[1] 1788559594.694497 punch from 0x47ef8c to 0x486e49 (0x7ebe bytes) 000171[0] 1788559594.828642 write 0x0fe000 thru 0x102ab3 (0x4ab4 bytes) 000172[0] 1788559594.884514 write 0x03d000 thru 0x04387b (0x687c bytes) 000173[0] 1788559594.950217 punch from 0x11729e to 0x1219bc (0xa71f bytes) 000179[1] 1788559595.082087 write_OD 0x4d0000 thru 0x4dcfff (0xd000 bytes) 000180[0] 1788559595.130687 mapwrite 0x56e000 thru 0x56eead (0xeae bytes) 000182[1] 1788559595.348135 write_OD 0x278000 thru 0x27bfff (0x4000 bytes) 000183[0] 1788559595.426017 punch from 0x08d16a to 0x091d92 (0x4c29 bytes) 000184[0] 1788559595.442894 mapwrite 0x514000 thru 0x514b43 (0xb44 bytes) 000185[0] 1788559595.506466 trunc from 0x56eeae to 0x9bca5e (0x44dbb1 bytes) 000187[1] 1788559595.555585 trunc from 0x9bca5f to 0x41b170 (0x5a18ee bytes) 000188[0] 1788559595.603291 mapwrite 0x23c000 thru 0x24957e (0xd57f bytes) 000189[1] 1788559595.759340 read_OD 0x15b000 thru 0x161fff (0x7000 bytes) 000190[1] 1788559595.787061 read_OD 0x31d000 thru 0x31dfff (0x1000 bytes) 000196[1] 1788559595.803299 write_OD 0x5dc000 thru 0x5e7fff (0xc000 bytes) 000199[0] 1788559595.884098 mapread 0x43f000 thru 0x44bc48 (0xcc49 bytes) 000200[0] 1788559595.998453 write 0x584000 thru 0x584826 (0x827 bytes) 000201[1] 1788559596.085090 trunc from 0x5e8000 to 0x7bc094 (0x1d4095 bytes) 000205[0] 1788559596.240903 write 0x7dc000 thru 0x7dca78 (0xa79 bytes) 000206[0] 1788559596.267049 write 0x32a000 thru 0x33085d (0x685e bytes) 000210[1] 1788559596.333895 write_OD 0x8c9000 thru 0x8cdfff (0x5000 bytes) 000211[1] 1788559596.489780 trunc from 0x8ce000 to 0x94b295 (0x7d296 bytes) 000213[1] 1788559596.553241 write_OD 0x5d7000 thru 0x5ddfff (0x7000 bytes) 000214[0] 1788559596.722409 read 0x172000 thru 0x17b638 (0x9639 bytes) 000215[1] 1788559596.758875 read_OD 0x42c000 thru 0x42ffff (0x4000 bytes) 000216[1] 1788559596.771939 write_OD 0x7f1000 thru 0x7f7fff (0x7000 bytes) 000217[0] 1788559596.920884 mapwrite 0x80b000 thru 0x8188f5 (0xd8f6 bytes) 000218[0] 1788559597.080199 write 0x142000 thru 0x14d274 (0xb275 bytes) 000219[1] 1788559597.093364 punch from 0x5acf23 to 0x5b46f7 (0x77d5 bytes) 000221[0] 1788559597.148344 mapwrite 0x0b3000 thru 0x0b688b (0x388c bytes) 000222[0] 1788559597.243283 read 0x22b000 thru 0x2338a5 (0x88a6 bytes) 000223[1] 1788559597.270708 mapread 0x669000 thru 0x672231 (0x9232 bytes) 000224[0] 1788559597.323976 read 0x0d7000 thru 0x0e1520 (0xa521 bytes) 000225[0] 1788559597.347216 mapwrite 0x394000 thru 0x394e11 (0xe12 bytes) 000226[0] 1788559597.489725 mapread 0x573000 thru 0x577a0f (0x4a10 bytes) 000227[0] 1788559597.531025 read 0x228000 thru 0x237cf6 (0xfcf7 bytes) 000229[1] 1788559597.548203 mapwrite 0x6a0000 thru 0x6a9aac (0x9aad bytes) 000230[1] 1788559597.691986 read_OD 0x3bb000 thru 0x3bffff (0x5000 bytes) 000231[0] 1788559597.734678 write 0x10b000 thru 0x10f665 (0x4666 bytes) 000232[1] 1788559597.765617 punch from 0x59509d to 0x59c563 (0x74c7 bytes) 000235[0] 1788559597.822461 trunc from 0x94b296 to 0x18db73 (0x7bd722 bytes) 000237[0] 1788559597.886942 write 0x729000 thru 0x734d72 (0xbd73 bytes) 000240[0] 1788559597.909871 mapread 0x6ee000 thru 0x6f4217 (0x6218 bytes) 000241[0] 1788559597.927095 mapwrite 0x705000 thru 0x711784 (0xc785 bytes) 000242[0] 1788559598.051465 write 0x981000 thru 0x98a914 (0x9915 bytes) 000243[0] 1788559598.073957 mapread 0x726000 thru 0x726637 (0x638 bytes) 000245[0] 1788559598.086499 punch from 0x977f89 to 0x97b109 (0x3181 bytes) 000246[1] 1788559598.122689 write_OD 0x0bb000 thru 0x0befff (0x4000 bytes) 000249[0] 1788559598.212047 read 0x85a000 thru 0x861bbd (0x7bbe bytes) 000250[0] 1788559598.226020 punch from 0x599a3e to 0x5a50c7 (0xb68a bytes) 000251[1] 1788559598.259372 mapwrite 0x498000 thru 0x4a52ae (0xd2af bytes) 000252[1] 1788559598.445325 write_OD 0x992000 thru 0x999fff (0x8000 bytes) 000253[1] 1788559598.534397 mapwrite 0x670000 thru 0x679076 (0x9077 bytes) 000254[1] 1788559598.679792 punch from 0x7307b9 to 0x735b21 (0x5369 bytes) 000255[0] 1788559598.698754 mapread 0x31d000 thru 0x323440 (0x6441 bytes) 000257[1] 1788559598.753541 trunc from 0x99a000 to 0x7a3f5c (0x1f60a3 bytes) 000258[1] 1788559598.807004 mapwrite 0x837000 thru 0x843d34 (0xcd35 bytes) 000260[0] 1788559599.057095 punch from 0x51b7fc to 0x5210a1 (0x58a6 bytes) 000261[1] 1788559599.092599 write_OD 0x8b7000 thru 0x8c3fff (0xd000 bytes) 000262[0] 1788559599.212875 punch from 0x4a6f83 to 0x4af609 (0x8687 bytes) 000263[1] 1788559599.283929 write_OD 0x2ba000 thru 0x2c1fff (0x8000 bytes) 000265[0] 1788559599.425471 punch from 0x0de29a to 0x0e2a1f (0x4786 bytes) 000266[0] 1788559599.466792 mapread 0x28f000 thru 0x293727 (0x4728 bytes) 000269[1] 1788559599.522519 read_OD 0x641000 thru 0x64cfff (0xc000 bytes) 000270[0] 1788559599.576039 read 0x1ba000 thru 0x1bfdfe (0x5dff bytes) 000271[1] 1788559599.588511 mapread 0x676000 thru 0x67bf89 (0x5f8a bytes) 000272[0] 1788559599.606619 mapread 0x4e9000 thru 0x4ebfc7 (0x2fc8 bytes) 000273[1] 1788559599.637698 write_OD 0x807000 thru 0x811fff (0xb000 bytes) 000274[0] 1788559599.695984 trunc from 0x8c4000 to 0x3db7e1 (0x4e881e bytes) 000275[1] 1788559599.770107 mapread 0x33b000 thru 0x33c096 (0x1097 bytes) 000276[1] 1788559599.810975 write_OD 0x624000 thru 0x624fff (0x1000 bytes) 000278[1] 1788559599.905001 read_OD 0x352000 thru 0x359fff (0x8000 bytes) 000279[0] 1788559599.918982 read 0x0f2000 thru 0x0f6e0f (0x4e10 bytes) 000280[1] 1788559599.930976 write_OD 0x356000 thru 0x357fff (0x2000 bytes) 000281[1] 1788559600.012210 read_OD 0x3e4000 thru 0x3e7fff (0x4000 bytes) 000282[1] 1788559600.027323 punch from 0x5b3774 to 0x5bcedf (0x976c bytes) 000283[0] 1788559600.066591 read 0x293000 thru 0x2a173f (0xe740 bytes) 000286[0] 1788559600.107399 mapread 0x2dd000 thru 0x2e4f97 (0x7f98 bytes) 000287[0] 1788559600.128588 write 0x715000 thru 0x721845 (0xc846 bytes) 000288[0] 1788559600.151206 trunc from 0x721846 to 0x708b48 (0x18cfd bytes) 000290[0] 1788559600.228384 mapwrite 0x1e1000 thru 0x1ed9b6 (0xc9b7 bytes) 000291[0] 1788559600.322657 mapread 0x55d000 thru 0x56666a (0x966b bytes) 000292[1] 1788559600.351946 mapread 0x50e000 thru 0x5153f9 (0x73fa bytes) 000293[1] 1788559600.389912 trunc from 0x708b49 to 0x1d4966 (0x5341e2 bytes) 000294[1] 1788559600.467023 mapread 0x16d000 thru 0x1793fa (0xc3fb bytes) 000295[0] 1788559600.516367 read 0x07c000 thru 0x07d3d5 (0x13d6 bytes) 000296[1] 1788559600.698558 mapwrite 0x0c9000 thru 0x0d3b79 (0xab7a bytes) 000299[1] 1788559601.026093 mapwrite 0x932000 thru 0x934d91 (0x2d92 bytes) 000300[1] 1788559601.271273 read_OD 0x697000 thru 0x698fff (0x2000 bytes) 000301[0] 1788559601.300221 mapread 0x678000 thru 0x6782d3 (0x2d4 bytes) 000303[1] 1788559601.485374 read_OD 0x707000 thru 0x70ffff (0x9000 bytes) 000305[1] 1788559601.513463 trunc from 0x934d92 to 0x275ca3 (0x6bf0ee bytes) 000306[0] 1788559601.608102 mapread 0x14d000 thru 0x14fafb (0x2afc bytes) 000309[0] 1788559601.647828 mapwrite 0x6bb000 thru 0x6c468c (0x968d bytes) 000310[0] 1788559601.907362 read 0x0f4000 thru 0x0f6d3e (0x2d3f bytes) 000312[1] 1788559601.963934 punch from 0x630f62 to 0x6335f9 (0x2698 bytes) 000314[1] 1788559602.025097 mapread 0x112000 thru 0x116ee1 (0x4ee2 bytes) 000316[1] 1788559602.111692 mapread 0x5a9000 thru 0x5b671f (0xd720 bytes) 000317[1] 1788559602.153977 write_OD 0x555000 thru 0x556fff (0x2000 bytes) 000319[0] 1788559602.283533 mapread 0x085000 thru 0x08cd0a (0x7d0b bytes) 000320[0] 1788559602.353560 trunc from 0x6c468d to 0x27d33f (0x44734d bytes) 000322[1] 1788559602.398306 trunc from 0x27d340 to 0x3d43fe (0x1570bf bytes) 000323[1] 1788559602.433836 mapread 0x286000 thru 0x2952be (0xf2bf bytes) 000324[1] 1788559602.474110 punch from 0x056d8f to 0x05b83f (0x4ab1 bytes) 000329[1] 1788559602.536772 mapwrite 0x5bf000 thru 0x5c9987 (0xa988 bytes) 000331[0] 1788559602.782882 read 0x1e5000 thru 0x1f1ec9 (0xceca bytes) 000333[0] 1788559602.809754 write 0x8a1000 thru 0x8ad14f (0xc150 bytes) 000336[1] 1788559602.847637 read_OD 0x5b6000 thru 0x5b9fff (0x4000 bytes) 000337[0] 1788559602.912297 read 0x654000 thru 0x65c3f2 (0x83f3 bytes) 000339[1] 1788559602.923838 write_OD 0x9ea000 thru 0x9f1fff (0x8000 bytes) 000340[1] 1788559603.032233 write_OD 0x5c2000 thru 0x5cafff (0x9000 bytes) 000343[1] 1788559603.085032 trunc from 0x9f2000 to 0x48e59a (0x563a65 bytes) 000346[0] 1788559603.223979 mapread 0x207000 thru 0x213f26 (0xcf27 bytes) 000353[1] 1788559603.318088 mapwrite 0x704000 thru 0x707e6d (0x3e6e bytes) 000354[1] 1788559603.428027 mapread 0x526000 thru 0x5271d1 (0x11d2 bytes) 000355[1] 1788559603.442924 punch from 0x220eaa to 0x22cdd3 (0xbf2a bytes) 000356[0] 1788559603.450948 trunc from 0x707e6e to 0x6dae1d (0x2d050 bytes) 000358[0] 1788559603.544385 punch from 0x03ccb4 to 0x049234 (0xc581 bytes) 000360[1] 1788559603.573459 read_OD 0x5a7000 thru 0x5aefff (0x8000 bytes) truncating to largest ever: 0x9fde92 000361[1] 1788559603.638819 trunc from 0x6dae1e to 0x9fde91 (0x323074 bytes) 000362[0] 1788559603.707627 trunc from 0x9fde92 to 0x12c4fe (0x8d1993 bytes) 000365[1] 1788559603.791409 read_OD 0x020000 thru 0x02efff (0xf000 bytes) 000368[1] 1788559603.879825 mapread 0x039000 thru 0x03c793 (0x3794 bytes) 000369[0] 1788559603.936904 read 0x00b000 thru 0x015d11 (0xad12 bytes) 000372[1] 1788559603.984831 punch from 0x0a8275 to 0x0af9ba (0x7746 bytes) 000373[1] 1788559604.043580 trunc from 0x12c4ff to 0x189f88 (0x5da8a bytes) 000374[1] 1788559604.123619 mapwrite 0x817000 thru 0x817ba8 (0xba9 bytes) 000376[0] 1788559604.305287 write 0x0c8000 thru 0x0c9372 (0x1373 bytes) 000377[0] 1788559604.407575 mapread 0x364000 thru 0x36c05c (0x805d bytes) 000379[1] 1788559604.413463 punch from 0x5bedeb to 0x5c0606 (0x181c bytes) 000380[1] 1788559604.440610 mapread 0x1a8000 thru 0x1b7a08 (0xfa09 bytes) 000381[1] 1788559604.524908 write_OD 0x615000 thru 0x621fff (0xd000 bytes) 000382[1] 1788559604.620841 mapread 0x486000 thru 0x48a96f (0x4970 bytes) 000383[0] 1788559604.652909 mapwrite 0x391000 thru 0x396f26 (0x5f27 bytes) 000385[0] 1788559604.772718 write 0x353000 thru 0x360862 (0xd863 bytes) 000386[0] 1788559604.780207 mapread 0x42e000 thru 0x434b6b (0x6b6c bytes) 000389[0] 1788559604.872139 read 0x01d000 thru 0x0256e7 (0x86e8 bytes) 000392[1] 1788559604.935849 read_OD 0x75e000 thru 0x767fff (0xa000 bytes) 000395[1] 1788559605.005975 mapread 0x2a8000 thru 0x2b5e76 (0xde77 bytes) 000396[1] 1788559605.052800 read_OD 0x4b9000 thru 0x4bafff (0x2000 bytes) 000397[0] 1788559605.062617 punch from 0x4c02c6 to 0x4cd49a (0xd1d5 bytes) 000399[1] 1788559605.158009 write_OD 0x14e000 thru 0x154fff (0x7000 bytes) 000400[0] 1788559605.238551 trunc from 0x817ba9 to 0x319931 (0x4fe277 bytes) 000404[0] 1788559605.298259 punch from 0x2bd3f5 to 0x2c399a (0x65a6 bytes) 000405[1] 1788559605.315441 trunc from 0x319932 to 0x61131e (0x2f79ed bytes) 000408[1] 1788559605.400888 mapwrite 0x4b1000 thru 0x4bbbfa (0xabfb bytes) 000412[0] 1788559605.498110 mapread 0x520000 thru 0x52ca5b (0xca5c bytes) 000413[1] 1788559605.569416 punch from 0x36f933 to 0x3787c9 (0x8e97 bytes) 000415[0] 1788559605.597889 mapwrite 0x924000 thru 0x92a60e (0x660f bytes) 000418[1] 1788559605.744130 punch from 0x1745c0 to 0x178b93 (0x45d4 bytes) 000419[1] 1788559605.807180 mapread 0x2df000 thru 0x2e0e27 (0x1e28 bytes) 000421[0] 1788559605.831903 trunc from 0x92a60f to 0x9106dd (0x19f31 bytes) 000424[1] 1788559605.942948 punch from 0x4e8b0a to 0x4f0479 (0x7970 bytes) 000425[0] 1788559605.978207 punch from 0x14487e to 0x1468de (0x2061 bytes) 000426[0] 1788559606.009025 punch from 0x1cd000 to 0x1d67fb (0x97fc bytes) 000427[1] 1788559606.027881 read_OD 0x162000 thru 0x166fff (0x5000 bytes) 000428[1] 1788559606.054319 write_OD 0x37c000 thru 0x37efff (0x3000 bytes) 000430[1] 1788559606.110878 mapwrite 0x44b000 thru 0x457f13 (0xcf14 bytes) 000431[1] 1788559606.205755 trunc from 0x9106de to 0x3ecd94 (0x523949 bytes) 000432[0] 1788559606.232790 punch from 0x3dd714 to 0x3de2ab (0xb98 bytes) 000433[1] 1788559606.274956 trunc from 0x3ecd95 to 0x84017c (0x4533e8 bytes) 000439[0] 1788559606.412753 read 0x4f0000 thru 0x4fb41c (0xb41d bytes) 000440[1] 1788559606.440902 mapread 0x7e5000 thru 0x7ea54e (0x554f bytes) 000441[1] 1788559606.475444 punch from 0x4780c3 to 0x487ffb (0xff39 bytes) 000442[0] 1788559606.526412 write 0x37b000 thru 0x37df9f (0x2fa0 bytes) 000443[1] 1788559606.554566 mapwrite 0x835000 thru 0x841d9a (0xcd9b bytes) 000444[1] 1788559606.735858 punch from 0x7f9225 to 0x8029b7 (0x9793 bytes) 000445[1] 1788559606.799195 write_OD 0x638000 thru 0x643fff (0xc000 bytes) 000446[0] 1788559606.930257 mapwrite 0x54d000 thru 0x5542a2 (0x72a3 bytes) 000447[1] 1788559607.085401 punch from 0x05924f to 0x065cc3 (0xca75 bytes) 000449[1] 1788559607.126480 mapwrite 0x49a000 thru 0x49e10a (0x410b bytes) 000453[0] 1788559607.379111 read 0x63d000 thru 0x647b1a (0xab1b bytes) 000454[0] 1788559607.454040 mapwrite 0x77e000 thru 0x78adea (0xcdeb bytes) 000455[0] 1788559607.734548 punch from 0x7a0ea3 to 0x7a6472 (0x55d0 bytes) 000456[1] 1788559607.771451 trunc from 0x841d9b to 0x63435b (0x20da3f bytes) 000458[1] 1788559607.847568 mapread 0x18f000 thru 0x19867d (0x967e bytes) 000461[0] 1788559607.929551 punch from 0x5acf46 to 0x5b389e (0x6959 bytes) 000462[1] 1788559607.982006 punch from 0x3bdd25 to 0x3c5279 (0x7555 bytes) 000463[0] 1788559608.040561 read 0x265000 thru 0x26b6c0 (0x66c1 bytes) 000469[1] 1788559608.105773 trunc from 0x63435c to 0x214356 (0x420005 bytes) 000472[0] 1788559608.214062 read 0x00a000 thru 0x00fd65 (0x5d66 bytes) 000473[1] 1788559608.297849 punch from 0x16ece6 to 0x16ff67 (0x1282 bytes) 000475[0] 1788559608.352497 read 0x20b000 thru 0x20defc (0x2efd bytes) 000476[0] 1788559608.421613 punch from 0x17c167 to 0x1846ab (0x8545 bytes) 000477[0] 1788559608.463942 read 0x02c000 thru 0x03b302 (0xf303 bytes) 000478[0] 1788559608.584877 punch from 0x169f1f to 0x170e65 (0x6f47 bytes) 000480[1] 1788559608.618946 trunc from 0x214357 to 0x564978 (0x350622 bytes) 000482[1] 1788559608.732219 trunc from 0x564979 to 0x79ad1d (0x2363a5 bytes) 000485[0] 1788559608.779092 punch from 0x212bbc to 0x21fe63 (0xd2a8 bytes) 000487[1] 1788559608.862706 read_OD 0x509000 thru 0x516fff (0xe000 bytes) 000490[1] 1788559608.953486 trunc from 0x79ad1e to 0x3cfc82 (0x3cb09b bytes) 000492[1] 1788559609.065382 mapread 0x18f000 thru 0x190a60 (0x1a61 bytes) 000494[1] 1788559609.131896 trunc from 0x3cfc83 to 0x85924e (0x4895cc bytes) 000495[1] 1788559609.172967 mapwrite 0x1f1000 thru 0x1fe244 (0xd245 bytes) 000496[0] 1788559609.252948 punch from 0x552b47 to 0x56104d (0xe507 bytes) 000498[1] 1788559609.309241 mapread 0x133000 thru 0x1379f3 (0x49f4 bytes) 000499[1] 1788559609.341800 mapwrite 0x824000 thru 0x82c3b0 (0x83b1 bytes) All operations completed A-OK! PASS 16a (72s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 18:06:57 (1788559617) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.478648 s, 17.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.460463 s, 18.2 MB/s Chance of close/open is 1 in 50 Seed set to 9622 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000006[1] 1788559623.352745 write 0x197cb8 thru 0x1983b2 (0x6fb bytes) 000008[0] 1788559623.379647 read 0x0cf518 thru 0x0d06b4 (0x119d bytes) 000009[0] 1788559623.415280 punch from 0x192de9 to 0x194459 (0x1671 bytes) 000010[1] 1788559623.445723 read 0x14f92e thru 0x14f97a (0x4d bytes) truncating to largest ever: 0x42a8fb 000011[0] 1788559623.498253 trunc from 0x1983b3 to 0x42a8fa (0x292548 bytes) truncating to largest ever: 0x6ee15c 000013[1] 1788559623.602729 trunc from 0x42a8fb to 0x6ee15b (0x2c3861 bytes) 000014[0] 1788559623.649134 read 0x6e7dec thru 0x6e970a (0x191f bytes) 000016[0] 1788559623.707547 write 0x377547 thru 0x3786b3 (0x116d bytes) 000017[1] 1788559623.757398 read 0x14771f thru 0x148cbe (0x15a0 bytes) 000018[1] 1788559623.800766 write 0x00c815 thru 0x00d961 (0x114d bytes) 000019[1] 1788559623.830533 mapwrite 0x6310ea thru 0x632414 (0x132b bytes) 000020[0] 1788559623.914381 punch from 0x6a8d7b to 0x6a9ea2 (0x1128 bytes) 000021[1] 1788559623.948736 mapread 0x476988 thru 0x477589 (0xc02 bytes) 000022[1] 1788559624.014432 mapread 0x4518e6 thru 0x451946 (0x61 bytes) 000023[0] 1788559624.124674 write 0x548db3 thru 0x549af9 (0xd47 bytes) 000024[1] 1788559624.205047 mapwrite 0x708c80 thru 0x709c45 (0xfc6 bytes) 000025[0] 1788559624.365599 write 0x1e0878 thru 0x1e1840 (0xfc9 bytes) 000026[0] 1788559624.417519 mapwrite 0x6193e8 thru 0x619850 (0x469 bytes) 000027[0] 1788559624.506477 punch from 0x2e5837 to 0x2e6f3f (0x1709 bytes) 000029[1] 1788559624.626355 write 0x75cc93 thru 0x75d741 (0xaaf bytes) 000030[0] 1788559624.663282 write 0x3674fe thru 0x368a7e (0x1581 bytes) 000033[0] 1788559624.716968 read 0x17bb3e thru 0x17c98b (0xe4e bytes) 000034[0] 1788559624.750332 mapwrite 0x1039b3 thru 0x104f9f (0x15ed bytes) 000036[0] 1788559624.884593 read 0x3744d2 thru 0x374d21 (0x850 bytes) 000039[1] 1788559624.992620 read 0x588bf5 thru 0x589cfa (0x1106 bytes) 000040[0] 1788559625.078123 trunc from 0x75d742 to 0x33c0f9 (0x421648 bytes) 000041[0] 1788559625.151597 read 0x22d1c8 thru 0x22d537 (0x370 bytes) 000042[1] 1788559625.286263 write 0x918f29 thru 0x919671 (0x749 bytes) 000045[0] 1788559625.371770 write 0x372906 thru 0x372bb3 (0x2ae bytes) 000046[1] 1788559625.462722 punch from 0x12adc6 to 0x12c16d (0x13a8 bytes) 000047[1] 1788559625.531128 mapwrite 0x021187 thru 0x022207 (0x1081 bytes) 000049[1] 1788559625.679497 mapread 0x0d9571 thru 0x0da2c7 (0xd57 bytes) 000051[1] 1788559625.754887 mapwrite 0x2bd7ea thru 0x2beda7 (0x15be bytes) 000053[1] 1788559625.903896 read 0x77111b thru 0x77285a (0x1740 bytes) 000055[1] 1788559625.957140 mapread 0x89645e thru 0x89689d (0x440 bytes) 000060[1] 1788559625.991862 mapread 0x480244 thru 0x481749 (0x1506 bytes) 000063[0] 1788559626.046468 mapwrite 0x1eb213 thru 0x1eccf9 (0x1ae7 bytes) 000064[1] 1788559626.135504 mapread 0x150c9b thru 0x152023 (0x1389 bytes) 000067[0] 1788559626.180787 mapwrite 0x8c48d0 thru 0x8c51ac (0x8dd bytes) 000069[1] 1788559626.329732 read 0x24fb78 thru 0x251855 (0x1cde bytes) 000070[0] 1788559626.358989 mapwrite 0x010db4 thru 0x010f31 (0x17e bytes) 000071[1] 1788559626.519144 mapread 0x204f56 thru 0x206300 (0x13ab bytes) 000073[1] 1788559626.598007 read 0x48f13e thru 0x49098f (0x1852 bytes) 000077[1] 1788559626.608551 mapread 0x55f969 thru 0x560354 (0x9ec bytes) 000081[0] 1788559626.623001 write 0x98c636 thru 0x98d916 (0x12e1 bytes) 000082[1] 1788559626.636402 mapwrite 0x395dae thru 0x396fe7 (0x123a bytes) 000083[1] 1788559626.733856 read 0x4aa1b5 thru 0x4ab09e (0xeea bytes) 000085[0] 1788559626.756585 write 0x71a465 thru 0x71c05b (0x1bf7 bytes) 000088[1] 1788559626.847257 punch from 0x961389 to 0x962945 (0x15bd bytes) 000089[0] 1788559626.876031 write 0x5785ab thru 0x57965f (0x10b5 bytes) 000091[0] 1788559626.913552 mapread 0x4df2fd thru 0x4df893 (0x597 bytes) 000092[1] 1788559626.962748 write 0x61259f thru 0x612e9d (0x8ff bytes) 000093[0] 1788559627.014939 read 0x8db5ae thru 0x8dbdba (0x80d bytes) truncating to largest ever: 0x960066 000094[1] 1788559627.071039 trunc from 0x98d917 to 0x960065 (0x2d8b1 bytes) 000097[0] 1788559627.223866 write 0x5cc43a thru 0x5cd254 (0xe1b bytes) 000098[1] 1788559627.272243 mapwrite 0x31a90f thru 0x31b79a (0xe8c bytes) 000099[1] 1788559627.356477 mapread 0x10940b thru 0x10a982 (0x1578 bytes) truncating to largest ever: 0x9879fd 000104[1] 1788559627.496515 trunc from 0x960066 to 0x9879fc (0x27997 bytes) 000105[0] 1788559627.590956 trunc from 0x9879fd to 0x45630d (0x5316ef bytes) 000107[0] 1788559627.646053 punch from 0x43e0f2 to 0x43e9b1 (0x8c0 bytes) 000108[1] 1788559627.652576 mapwrite 0x8917e8 thru 0x8932ea (0x1b03 bytes) 000109[0] 1788559627.771594 punch from 0x0d0ab9 to 0x0d1e2f (0x1377 bytes) 000110[1] 1788559627.807740 mapread 0x18d7d1 thru 0x18e183 (0x9b3 bytes) 000111[0] 1788559627.860934 write 0x509407 thru 0x50981a (0x414 bytes) 000112[0] 1788559627.907848 mapwrite 0x6ee73e thru 0x6efbbf (0x1482 bytes) 000113[1] 1788559628.008723 mapwrite 0x1963f6 thru 0x197615 (0x1220 bytes) 000114[0] 1788559628.141584 read 0x1ebe7c thru 0x1ebff7 (0x17c bytes) 000115[1] 1788559628.177434 read 0x2c285c thru 0x2c3304 (0xaa9 bytes) 000116[0] 1788559628.183467 read 0x712162 thru 0x712192 (0x31 bytes) 000118[1] 1788559628.210028 write 0x211738 thru 0x21354e (0x1e17 bytes) 000119[0] 1788559628.247292 read 0x054d66 thru 0x054fcd (0x268 bytes) 000120[1] 1788559628.269915 read 0x1065a1 thru 0x107203 (0xc63 bytes) 000121[0] 1788559628.279101 punch from 0x58a99d to 0x58bb66 (0x11ca bytes) 000125[0] 1788559628.304183 trunc from 0x8932eb to 0x728e2f (0x16a4bb bytes) 000127[1] 1788559628.372293 mapwrite 0x74f4aa thru 0x750d01 (0x1858 bytes) 000128[1] 1788559628.626401 mapread 0x6509f6 thru 0x652723 (0x1d2e bytes) 000129[0] 1788559628.708305 read 0x179160 thru 0x1794f8 (0x399 bytes) 000130[1] 1788559628.753798 mapread 0x4e49d7 thru 0x4e5e52 (0x147c bytes) 000131[0] 1788559628.778379 read 0x3f3835 thru 0x3f513a (0x1906 bytes) 000132[1] 1788559628.783651 trunc from 0x750d02 to 0x2900ea (0x4c0c17 bytes) 000133[0] 1788559628.855710 write 0x3b5be7 thru 0x3b6460 (0x87a bytes) 000134[0] 1788559628.886216 write 0x0d28e9 thru 0x0d3430 (0xb48 bytes) 000137[1] 1788559628.955142 mapread 0x13c25b thru 0x13dd18 (0x1abe bytes) 000138[1] 1788559629.012721 trunc from 0x3b6461 to 0x840c08 (0x48a7a8 bytes) 000139[1] 1788559629.090633 write 0x577619 thru 0x5781cb (0xbb3 bytes) 000143[1] 1788559629.158245 mapwrite 0x8c41d9 thru 0x8c595d (0x1785 bytes) 000144[1] 1788559629.291525 mapwrite 0x80c900 thru 0x80e7dd (0x1ede bytes) 000145[1] 1788559629.370428 read 0x21ed8e thru 0x220b1e (0x1d91 bytes) 000147[1] 1788559629.458964 write 0x69720a thru 0x6991b2 (0x1fa9 bytes) 000148[1] 1788559629.478430 trunc from 0x8c595e to 0x5bec5d (0x306d00 bytes) 000149[0] 1788559629.597081 write 0x6ecf3c thru 0x6eea80 (0x1b45 bytes) 000151[0] 1788559629.713194 read 0x36f0f7 thru 0x36fa80 (0x98a bytes) 000155[1] 1788559629.792645 mapwrite 0x2d030e thru 0x2d2248 (0x1f3b bytes) 000156[1] 1788559629.939260 mapread 0x5a18f8 thru 0x5a389f (0x1fa8 bytes) 000158[0] 1788559630.043036 trunc from 0x6eea81 to 0x729b49 (0x3b0c9 bytes) 000160[0] 1788559630.135965 write 0x7514e7 thru 0x7522c6 (0xde0 bytes) 000161[0] 1788559630.185776 punch from 0x369d5c to 0x36b17d (0x1422 bytes) 000166[1] 1788559630.363129 mapread 0x139412 thru 0x13b310 (0x1eff bytes) 000168[0] 1788559630.445593 mapread 0x564fea thru 0x565f3e (0xf55 bytes) 000173[0] 1788559630.548068 mapwrite 0x2f51dd thru 0x2f5b37 (0x95b bytes) 000174[1] 1788559630.691844 mapwrite 0x24dea7 thru 0x24f251 (0x13ab bytes) 000175[1] 1788559630.816566 read 0x349e21 thru 0x34b2af (0x148f bytes) 000177[0] 1788559630.900370 punch from 0x27dd17 to 0x27ecfe (0xfe8 bytes) 000178[0] 1788559630.938685 punch from 0x100a8b to 0x102434 (0x19aa bytes) 000179[0] 1788559630.967600 punch from 0x246376 to 0x246847 (0x4d2 bytes) 000181[1] 1788559631.024595 read 0x6150f4 thru 0x616506 (0x1413 bytes) 000183[1] 1788559631.131945 read 0x2454d6 thru 0x245fe7 (0xb12 bytes) 000184[1] 1788559631.155793 punch from 0x11bbb5 to 0x11d93f (0x1d8b bytes) 000185[0] 1788559631.266299 mapwrite 0x985d89 thru 0x9871c0 (0x1438 bytes) 000186[0] 1788559631.525124 write 0x81c3cb thru 0x81d6fd (0x1333 bytes) 000187[1] 1788559631.531012 read 0x8df90f thru 0x8e0bcd (0x12bf bytes) 000189[1] 1788559631.682905 mapread 0x1b72ee thru 0x1b7917 (0x62a bytes) 000190[0] 1788559631.738719 write 0x48e5c9 thru 0x48f3d7 (0xe0f bytes) 000193[1] 1788559631.799237 mapwrite 0x718c96 thru 0x71abee (0x1f59 bytes) 000194[1] 1788559631.967571 punch from 0x1b60e8 to 0x1b632d (0x246 bytes) 000195[0] 1788559632.020528 mapread 0x984f6b thru 0x986e9f (0x1f35 bytes) 000196[0] 1788559632.092343 read 0x4314a6 thru 0x431721 (0x27c bytes) 000197[1] 1788559632.125788 read 0x21c1cf thru 0x21dc2c (0x1a5e bytes) 000198[1] 1788559632.135060 punch from 0x8a9b1f to 0x8ab78f (0x1c71 bytes) 000199[1] 1788559632.203958 read 0x3fef6a thru 0x40052d (0x15c4 bytes) 000202[1] 1788559632.230571 mapread 0x64aae9 thru 0x64c422 (0x193a bytes) 000203[1] 1788559632.249394 write 0x34bf48 thru 0x34c1ae (0x267 bytes) 000207[0] 1788559632.281578 write 0x2b68be thru 0x2b782a (0xf6d bytes) 000210[0] 1788559632.361464 write 0x08fe47 thru 0x0909bd (0xb77 bytes) 000211[0] 1788559632.398064 mapwrite 0x7bd038 thru 0x7be102 (0x10cb bytes) 000213[0] 1788559632.467422 read 0x55443b thru 0x5551af (0xd75 bytes) 000214[0] 1788559632.497211 punch from 0x27f1d0 to 0x2811a9 (0x1fda bytes) 000215[0] 1788559632.517992 mapwrite 0x819db2 thru 0x81af43 (0x1192 bytes) 000216[0] 1788559632.582757 write 0x05fb1b thru 0x061631 (0x1b17 bytes) 000217[1] 1788559632.638554 write 0x325932 thru 0x3263b7 (0xa86 bytes) 000218[0] 1788559632.677959 read 0x537344 thru 0x539215 (0x1ed2 bytes) 000219[1] 1788559632.699904 mapwrite 0x72055d thru 0x720669 (0x10d bytes) 000220[1] 1788559632.776405 punch from 0x9371c6 to 0x938dac (0x1be7 bytes) 000221[1] 1788559632.804245 read 0x4467e2 thru 0x4471e9 (0xa08 bytes) 000223[1] 1788559632.811995 read 0x260067 thru 0x261160 (0x10fa bytes) 000224[1] 1788559632.820324 read 0x67bda9 thru 0x67ce83 (0x10db bytes) 000225[0] 1788559632.845968 mapwrite 0x0e0d14 thru 0x0e16cd (0x9ba bytes) 000226[1] 1788559632.942839 punch from 0x1e6222 to 0x1e7771 (0x1550 bytes) 000227[1] 1788559632.959656 mapread 0x7f1cee thru 0x7f389d (0x1bb0 bytes) 000228[1] 1788559632.998088 write 0x942810 thru 0x9445ec (0x1ddd bytes) 000230[0] 1788559633.014291 trunc from 0x9871c1 to 0x321954 (0x66586c bytes) 000232[0] 1788559633.099439 mapread 0x235fe5 thru 0x2377bb (0x17d7 bytes) 000233[0] 1788559633.119281 read 0x04748b thru 0x048fc6 (0x1b3c bytes) 000234[1] 1788559633.180947 trunc from 0x321955 to 0x21e443 (0x103511 bytes) 000235[0] 1788559633.273470 trunc from 0x21e444 to 0x5737cd (0x35538a bytes) 000237[1] 1788559633.446489 mapread 0x0b3e86 thru 0x0b5582 (0x16fd bytes) 000238[1] 1788559633.525742 write 0x28c6a8 thru 0x28d98b (0x12e4 bytes) 000239[1] 1788559633.557873 write 0x07f4b0 thru 0x08074d (0x129e bytes) 000240[0] 1788559633.586554 punch from 0x0a7f04 to 0x0a9187 (0x1284 bytes) 000241[0] 1788559633.614540 write 0x2f9e85 thru 0x2fb3c9 (0x1545 bytes) 000243[1] 1788559633.634703 read 0x55f462 thru 0x5602ee (0xe8d bytes) 000244[1] 1788559633.652597 mapread 0x109b9a thru 0x10a19a (0x601 bytes) 000245[1] 1788559633.700705 mapread 0x03e3c5 thru 0x03e771 (0x3ad bytes) 000247[0] 1788559633.715421 mapread 0x005f37 thru 0x006d67 (0xe31 bytes) 000248[0] 1788559633.724293 write 0x918be0 thru 0x91a9dd (0x1dfe bytes) 000249[1] 1788559633.748841 mapread 0x151265 thru 0x152ad4 (0x1870 bytes) 000252[1] 1788559633.803821 trunc from 0x91a9de to 0x40ef0f (0x50bace bytes) 000253[1] 1788559633.889383 mapwrite 0x860d28 thru 0x861c6b (0xf44 bytes) 000254[0] 1788559634.026913 write 0x8733c7 thru 0x873d6a (0x9a4 bytes) 000255[1] 1788559634.096555 trunc from 0x873d6b to 0x5809f0 (0x2f337a bytes) 000258[1] 1788559634.231385 mapwrite 0x86d96f thru 0x86e69d (0xd2f bytes) 000261[1] 1788559634.387090 trunc from 0x86e69e to 0x0d288e (0x79be0f bytes) 000262[0] 1788559634.437871 punch from 0x0226ad to 0x0238cd (0x1221 bytes) 000263[1] 1788559634.476706 trunc from 0x0d288f to 0x49f58a (0x3cccfc bytes) 000264[0] 1788559634.556357 mapwrite 0x8a38ef thru 0x8a4a16 (0x1128 bytes) 000267[1] 1788559634.914477 mapread 0x012017 thru 0x012ebf (0xea9 bytes) 000269[0] 1788559634.972584 punch from 0x6b90d4 to 0x6b97b4 (0x6e1 bytes) 000271[0] 1788559634.998942 mapread 0x3c2dc6 thru 0x3c4528 (0x1763 bytes) 000272[0] 1788559635.037355 trunc from 0x8a4a17 to 0x67a703 (0x22a313 bytes) 000273[0] 1788559635.096124 mapread 0x1f4e8d thru 0x1f5f19 (0x108d bytes) 000274[1] 1788559635.112931 mapread 0x2ef3c2 thru 0x2f0324 (0xf63 bytes) 000275[0] 1788559635.205881 write 0x38b92e thru 0x38cbcc (0x129f bytes) 000276[0] 1788559635.239658 punch from 0x043b6c to 0x043e91 (0x326 bytes) 000277[1] 1788559635.314982 read 0x02ba0b thru 0x02c164 (0x75a bytes) 000278[0] 1788559635.370520 write 0x32a904 thru 0x32aabd (0x1ba bytes) 000279[0] 1788559635.420870 punch from 0x2c3c72 to 0x2c4be1 (0xf70 bytes) 000280[0] 1788559635.453720 mapwrite 0x79c9f4 thru 0x79cf4b (0x558 bytes) 000281[0] 1788559635.531999 mapread 0x4230dd thru 0x423d7a (0xc9e bytes) 000282[1] 1788559635.559730 read 0x0d755c thru 0x0d913d (0x1be2 bytes) 000284[0] 1788559635.606893 mapread 0x1668a6 thru 0x1682a7 (0x1a02 bytes) 000286[1] 1788559635.691249 write 0x3a81ff thru 0x3a85a6 (0x3a8 bytes) 000288[0] 1788559635.784360 mapwrite 0x8f56cb thru 0x8f768c (0x1fc2 bytes) 000289[1] 1788559635.876208 read 0x78ae7a thru 0x78b8fd (0xa84 bytes) 000292[1] 1788559635.920437 trunc from 0x8f768d to 0x8889a3 (0x6ece9 bytes) 000293[0] 1788559635.990904 punch from 0x8111d3 to 0x812929 (0x1757 bytes) 000296[1] 1788559636.047416 mapwrite 0x52ce45 thru 0x52d694 (0x850 bytes) 000298[1] 1788559636.129483 trunc from 0x8889a4 to 0x9331dc (0xaa839 bytes) 000300[0] 1788559636.181482 punch from 0x4d0b5c to 0x4d1edd (0x1382 bytes) 000301[1] 1788559636.211997 mapread 0x074591 thru 0x07477d (0x1ed bytes) 000302[0] 1788559636.241284 mapread 0x126fbc thru 0x127cf2 (0xd37 bytes) 000304[1] 1788559636.316533 punch from 0x684509 to 0x684647 (0x13f bytes) 000305[0] 1788559636.335692 punch from 0x7d3a4d to 0x7d5527 (0x1adb bytes) 000308[0] 1788559636.381920 trunc from 0x9331dd to 0x932e75 (0x367 bytes) 000309[0] 1788559636.440725 mapread 0x2daecd thru 0x2dc707 (0x183b bytes) 000310[1] 1788559636.480940 mapread 0x716f3e thru 0x7184b5 (0x1578 bytes) 000312[1] 1788559636.551600 trunc from 0x932e76 to 0x6e3758 (0x24f71d bytes) 000313[0] 1788559636.612006 read 0x5e296e thru 0x5e386f (0xf02 bytes) 000314[0] 1788559636.647596 mapread 0x374500 thru 0x375416 (0xf17 bytes) 000315[1] 1788559636.674857 punch from 0x0bfab2 to 0x0c0e71 (0x13c0 bytes) 000316[0] 1788559636.705657 read 0x149992 thru 0x14b44f (0x1abe bytes) 000317[1] 1788559636.774930 mapread 0x11c863 thru 0x11e386 (0x1b24 bytes) 000318[0] 1788559636.803588 mapread 0x33dcf0 thru 0x33f2cc (0x15dd bytes) 000320[0] 1788559636.827724 mapwrite 0x5cc17d thru 0x5cc8a4 (0x728 bytes) 000325[1] 1788559636.965713 mapread 0x44becc thru 0x44c0a8 (0x1dd bytes) 000326[1] 1788559637.001460 mapread 0x040241 thru 0x041ae3 (0x18a3 bytes) 000327[1] 1788559637.023119 read 0x4d35e8 thru 0x4d3f0a (0x923 bytes) 000330[1] 1788559637.033397 read 0x2c5918 thru 0x2c733f (0x1a28 bytes) 000331[0] 1788559637.045825 trunc from 0x6e3759 to 0x7ddebb (0xfa763 bytes) 000333[1] 1788559637.136569 mapwrite 0x407ab2 thru 0x408cb3 (0x1202 bytes) 000335[0] 1788559637.250868 write 0x83606a thru 0x837307 (0x129e bytes) 000336[0] 1788559637.274118 read 0x6b46f4 thru 0x6b5e07 (0x1714 bytes) 000337[1] 1788559637.291315 punch from 0x287f75 to 0x28834b (0x3d7 bytes) 000338[0] 1788559637.312262 mapread 0x721bca thru 0x723abc (0x1ef3 bytes) truncating to largest ever: 0x9e94aa 000339[1] 1788559637.336656 trunc from 0x837308 to 0x9e94a9 (0x1b21a2 bytes) 000341[0] 1788559637.390522 write 0x4829fa thru 0x484034 (0x163b bytes) 000342[1] 1788559637.423428 mapwrite 0x25dcc9 thru 0x25e355 (0x68d bytes) 000343[1] 1788559637.489547 punch from 0x761c83 to 0x763285 (0x1603 bytes) 000344[0] 1788559637.538605 mapwrite 0x3a91f0 thru 0x3a9dde (0xbef bytes) 000345[0] 1788559637.650092 read 0x161a4b thru 0x162080 (0x636 bytes) 000348[1] 1788559637.798553 mapread 0x11e215 thru 0x11fe5c (0x1c48 bytes) 000349[1] 1788559637.861572 mapwrite 0x652de7 thru 0x652ebe (0xd8 bytes) 000352[1] 1788559637.935393 trunc from 0x9e94aa to 0x83abbb (0x1ae8ee bytes) 000353[0] 1788559637.974698 punch from 0x44baf0 to 0x44d29e (0x17af bytes) 000354[0] 1788559638.004985 read 0x11df34 thru 0x11f370 (0x143d bytes) 000355[1] 1788559638.029843 punch from 0x6f870d to 0x6f8808 (0xfc bytes) 000357[1] 1788559638.076401 mapread 0x0f90e4 thru 0x0fa612 (0x152f bytes) 000358[1] 1788559638.094721 mapread 0x4b3aa4 thru 0x4b3d22 (0x27f bytes) 000359[0] 1788559638.101488 mapread 0x65f3ca thru 0x65f88f (0x4c6 bytes) 000360[0] 1788559638.129249 punch from 0x1f5755 to 0x1f5f57 (0x803 bytes) 000361[1] 1788559638.162341 read 0x4a0f07 thru 0x4a2764 (0x185e bytes) 000362[1] 1788559638.169495 mapwrite 0x440b56 thru 0x441f6d (0x1418 bytes) 000364[0] 1788559638.328766 punch from 0x60175e to 0x602ac4 (0x1367 bytes) 000369[1] 1788559638.353355 mapwrite 0x77357e thru 0x773fc0 (0xa43 bytes) 000371[0] 1788559638.486783 trunc from 0x83abbc to 0x39b985 (0x49f236 bytes) 000373[1] 1788559638.520826 punch from 0x291c58 to 0x2931a0 (0x1549 bytes) 000375[0] 1788559638.608128 write 0x94a70e thru 0x94b1db (0xace bytes) 000377[0] 1788559638.699419 mapwrite 0x2930ef thru 0x294efa (0x1e0c bytes) 000379[0] 1788559638.819378 write 0x84201a thru 0x8435e4 (0x15cb bytes) 000381[0] 1788559638.850943 trunc from 0x94b1dc to 0x364a2a (0x5e67b1 bytes) 000382[0] 1788559638.896836 mapread 0x061900 thru 0x0635f2 (0x1cf3 bytes) 000383[1] 1788559638.941542 punch from 0x01563a to 0x0170de (0x1aa5 bytes) 000384[0] 1788559638.976687 trunc from 0x364a2b to 0x9a3db3 (0x63f389 bytes) 000388[0] 1788559639.041494 punch from 0x2c8e7a to 0x2c9159 (0x2e0 bytes) 000390[0] 1788559639.046952 trunc from 0x9a3db4 to 0x2bb090 (0x6e8d23 bytes) 000394[1] 1788559639.069999 mapread 0x0eecec thru 0x0f0b6d (0x1e82 bytes) 000398[0] 1788559639.115532 punch from 0x04610d to 0x0462a8 (0x19c bytes) 000401[1] 1788559639.213286 mapread 0x1e5269 thru 0x1e54ef (0x287 bytes) 000402[0] 1788559639.255465 mapread 0x298ee0 thru 0x29a764 (0x1885 bytes) 000403[1] 1788559639.274515 write 0x4b9e71 thru 0x4b9f52 (0xe2 bytes) 000404[1] 1788559639.303393 trunc from 0x4b9f53 to 0x3c3e1c (0xf6136 bytes) 000405[1] 1788559639.354446 mapwrite 0x0e90fc thru 0x0eae5d (0x1d62 bytes) 000406[0] 1788559639.448983 mapwrite 0x1d8a17 thru 0x1d9424 (0xa0e bytes) 000409[0] 1788559639.692705 mapwrite 0x6345ab thru 0x6360ff (0x1b55 bytes) 000410[1] 1788559639.919027 punch from 0x245e90 to 0x2467aa (0x91b bytes) 000412[0] 1788559639.992055 read 0x279683 thru 0x27b0a1 (0x1a1f bytes) 000413[0] 1788559640.031046 punch from 0x2d13f0 to 0x2d19b8 (0x5c9 bytes) 000416[1] 1788559640.141238 trunc from 0x636100 to 0x9dffca (0x3a9ecb bytes) 000417[1] 1788559640.243512 punch from 0x61aac8 to 0x61b94e (0xe87 bytes) 000420[0] 1788559640.277437 trunc from 0x9dffcb to 0x148df1 (0x8971d9 bytes) 000421[1] 1788559640.349587 mapwrite 0x4f878d thru 0x4fa130 (0x19a4 bytes) 000422[0] 1788559640.592877 read 0x186bcb thru 0x1888b5 (0x1ceb bytes) 000423[1] 1788559640.653701 punch from 0x016e4a to 0x018e02 (0x1fb9 bytes) 000426[1] 1788559640.757903 mapwrite 0x77306b thru 0x774ddc (0x1d72 bytes) 000427[0] 1788559640.842905 punch from 0x03d69e to 0x03e6c4 (0x1027 bytes) 000428[1] 1788559640.888559 mapread 0x0bab6e thru 0x0bc4a8 (0x193b bytes) 000429[0] 1788559640.945731 mapwrite 0x0fa249 thru 0x0fa8f2 (0x6aa bytes) 000436[1] 1788559641.167302 punch from 0x03d720 to 0x03e860 (0x1141 bytes) 000438[0] 1788559641.194043 punch from 0x081282 to 0x081d58 (0xad7 bytes) 000441[0] 1788559641.299575 write 0x5778c0 thru 0x578760 (0xea1 bytes) 000443[1] 1788559641.379326 mapwrite 0x2813f3 thru 0x281504 (0x112 bytes) 000444[0] 1788559641.482999 mapwrite 0x7305bd thru 0x73191a (0x135e bytes) 000445[0] 1788559641.568119 mapread 0x56060a thru 0x561626 (0x101d bytes) 000447[1] 1788559641.614760 punch from 0x5e6439 to 0x5e67db (0x3a3 bytes) 000449[0] 1788559641.655307 write 0x4c3869 thru 0x4c482a (0xfc2 bytes) 000450[0] 1788559641.688285 mapwrite 0x011592 thru 0x012e1d (0x188c bytes) 000451[0] 1788559641.765433 mapwrite 0x9b0279 thru 0x9b0459 (0x1e1 bytes) 000455[1] 1788559641.897028 punch from 0x27127a to 0x271975 (0x6fc bytes) 000456[0] 1788559641.935262 mapwrite 0x3b40c4 thru 0x3b43ce (0x30b bytes) 000460[1] 1788559642.083519 read 0x04224b thru 0x04224c (0x2 bytes) 000462[0] 1788559642.133477 read 0x3915c0 thru 0x3922af (0xcf0 bytes) 000463[0] 1788559642.143600 trunc from 0x9b045a to 0x762010 (0x24e449 bytes) 000466[1] 1788559642.210544 punch from 0x58b057 to 0x58cf6b (0x1f15 bytes) 000468[1] 1788559642.258833 mapread 0x0238b2 thru 0x02540c (0x1b5b bytes) 000469[0] 1788559642.298223 mapread 0x11fa04 thru 0x11fea8 (0x4a5 bytes) 000471[0] 1788559642.313144 punch from 0x4a102c to 0x4a22f0 (0x12c5 bytes) 000472[0] 1788559642.331823 read 0x4b452f thru 0x4b5507 (0xfd9 bytes) 000473[0] 1788559642.353437 mapread 0x2e85c1 thru 0x2e8e30 (0x870 bytes) 000474[0] 1788559642.372260 punch from 0x508724 to 0x508ed1 (0x7ae bytes) 000480[1] 1788559642.413588 trunc from 0x762011 to 0x604b21 (0x15d4ef bytes) 000481[0] 1788559642.496594 mapwrite 0x66d5ca thru 0x66da82 (0x4b9 bytes) 000482[1] 1788559642.581903 punch from 0x418538 to 0x419aaf (0x1578 bytes) 000483[1] 1788559642.621243 mapwrite 0x4ddb88 thru 0x4de7ed (0xc66 bytes) 000484[0] 1788559642.688097 write 0x320dfc thru 0x321b9c (0xda1 bytes) 000486[1] 1788559642.724212 write 0x823f1b thru 0x825ef5 (0x1fdb bytes) 000487[0] 1788559642.767459 mapwrite 0x888e27 thru 0x88a408 (0x15e2 bytes) 000488[1] 1788559642.951999 mapread 0x55df30 thru 0x55f901 (0x19d2 bytes) 000490[1] 1788559643.044904 mapread 0x0f02f5 thru 0x0f1890 (0x159c bytes) 000496[0] 1788559643.117052 punch from 0x01749a to 0x0176ab (0x212 bytes) 000497[0] 1788559643.133097 write 0x783e58 thru 0x783fcf (0x178 bytes) 000498[0] 1788559643.178261 read 0x13d0c1 thru 0x13eae8 (0x1a28 bytes) 000499[1] 1788559643.210896 mapread 0x536f74 thru 0x537357 (0x3e4 bytes) All operations completed A-OK! PASS 16b (32s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 18:07:29 (1788559649) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 18:07:34 (1788559654) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.483486 s, 17.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.408716 s, 20.5 MB/s ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff98f48547f000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff98f4902c5000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff98f4902c5000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff98f4902c5000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9101 s, 8.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.21592 s, 32.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.22518 s, 32.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.99688 s, 15.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.98303 s, 13.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.0823 s, 25.7 MB/s PASS 16d (52s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 18:08:26 (1788559706) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.325308 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157335 s, 6.7 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 18:08:35 (1788559715) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 18:09:03 (1788559743) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 18:09:32 (1788559772) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 18:09:41 (1788559781) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (9s) == sanityn test 16j: race dio with buffered i/o ========== 18:09:50 (1788559790) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.79238 s, 14.6 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.84555 s, 14.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.5043 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.2882 s, 11.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.37142 s, 7.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.79003 s, 9.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.12124 s, 6.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.77173 s, 9.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.87238 s, 9.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.38161 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.52126 s, 7.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.52207 s, 10.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.91504 s, 9.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.55175 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.75102 s, 7.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.83841 s, 6.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.5472 s, 5.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.10557 s, 5.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.22341 s, 5.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (49s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 18:10:39 (1788559839) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Sep 4 18:11:37 EDT 2026 + fsx 2 done: rc=0 Fri Sep 4 18:11:37 EDT 2026 + fsx 3 done: rc=0 Fri Sep 4 18:11:37 EDT 2026 + fsx 4 done: rc=0 Fri Sep 4 18:11:37 EDT 2026 + fsx 5 done: rc=0 Fri Sep 4 18:11:37 EDT 2026 PASS 16k (67s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 18:11:46 (1788559906) fail_loc=0x8000030a PASS 17 (11s) == sanityn test 18: mmap sanity check =========================================================================================== 18:11:57 (1788559917) mmap test1: basic mmap operation (PASS, 0.14747s) mmap test2: MAP_PRIVATE not write back (PASS, 0.16892s) mmap test3: concurrent mmap ops on two nodes (PASS, 18.135s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7772s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.21144s) mmap test6: check mmap write/read content on two nodes (PASS, 0.27032s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.175s) mmap test11: multi-thread shared mmap (PASS, 1.5588s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1355s) PASS 18 (37s) == sanityn test 19: test concurrent uncached read races ========================================================================= 18:12:34 (1788559954) oleg457-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (6s) == sanityn test 20: test extra readahead page left in cache ============================================================== 18:12:40 (1788559960) PASS 20 (12s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 18:12:52 (1788559972) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 18:13:01 (1788559981) now is 1788559985 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 new atime is 1788560046 PASS 23 (73s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 18:14:14 (1788560054) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 18:14:22 (1788560062) PASS 24b (8s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 18:14:30 (1788560070) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (10s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 18:14:40 (1788560080) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 18:14:45 (1788560085) utime: good mknod times 1788560085 <= 1788560086 <= 1788560086 for /mnt/lustre/f26a utime: good mknod times 1788560085 <= 1788560086 <= 1788560086 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (10s) == sanityn test 26b: sync mtime between ost and mds ====== 18:14:54 (1788560094) PASS 26b (15s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 18:15:10 (1788560110) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-04 18:15:14.000000000 -0400 Birth: 2026-09-04 18:15:14.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-04 18:15:14.000000000 -0400 Birth: 2026-09-04 18:15:14.000000000 -0400 PASS 26c (13s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 18:15:24 (1788560124) 0 PASS 30 (11s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 18:15:36 (1788560136) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 18:15:45 (1788560145) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7542784 39936 7497728 1% /mnt/lustre2 PASS 31b (25s) == sanityn test 31r: open-rename(replace) race =========== 18:16:11 (1788560171) -rw-r--r-- 1 root root 0 Sep 4 18:16 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 18:16:23 (1788560183) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43221 Terminated tail -f $tfile PASS 31s (11s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 18:16:33 (1788560193) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 18:16:43 (1788560203) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (6s) == sanityn test 32c: contention detection ================ 18:16:48 (1788560208) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.25709 s, 16.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.12632 s, 24.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.25307 s, 16.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.55759 s, 14.7 MB/s contention_seconds 4, contention_events 12 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.51865 s, 20.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.9494 s, 17.8 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.89331 s, 27.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.95494 s, 26.8 MB/s PASS 32c (39s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 18:17:29 (1788560249) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 18:17:32 (1788560252) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 18:17:36 (1788560256) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 18:17:40 (1788560260) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 18:17:44 (1788560264) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98f48547f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98f48547f000.ost_server_uuid in FULL state after 0 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98f48547f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98f48547f000.ost_server_uuid in IDLE state after 0 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98f48547f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98f48547f000.ost_server_uuid in IDLE state after 0 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98f48547f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98f48547f000.ost_server_uuid in IDLE state after 0 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98f48547f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98f48547f000.ost_server_uuid in IDLE state after 0 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98f48547f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff98f48547f000.ost_server_uuid in IDLE state after 0 sec PASS 34 (142s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 18:20:06 (1788560406) fail_loc=0x80000317 Race attempt 0 Wait for 50528 50549 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50528 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.157@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.157@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 620 (time 1788560426.02 total 10.02 last 61.87) - open/close 1208 (time 1788560436.03 total 20.02 last 58.78) - open/close 1818 (time 1788560446.03 total 30.03 last 60.95) - open/close 2440 (time 1788560456.05 total 40.05 last 62.08) - open/close 3080 (time 1788560466.06 total 50.06 last 63.96) - open/close 3664 (time 1788560476.07 total 60.06 last 58.36) total: 4000 open/close in 66.86 seconds: 59.83 ops/second PASS 35 (76s) == sanityn test 36: handle ESTALE/open-unlink correctly == 18:21:23 (1788560483) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.756009 s, 69.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7432192) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.855169 s, 61.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.811904 s, 64.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7489536) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.867004 s, 60.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7490560) after_dd(7424000) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.637061 s, 82.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7436288) after(7487488) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.715414 s, 73.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7487488) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.682807 s, 76.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7489536) after_dd(7424000) after(7475200) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.794023 s, 66.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7475200) after_dd(7435264) after(7486464) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53887 s, 34.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7486464) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.785865 s, 66.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(9) *** before(7496704) after_dd(7419904) after(7471104) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.946668 s, 55.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7471104) after_dd(7439360) after(7490560) PASS 36 (313s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 18:26:35 (1788560795) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 - create 1238 (time 1788560809.30 total 10.02 last 123.58) - create 2327 (time 1788560819.32 total 20.03 last 108.76) - create 3267 (time 1788560829.32 total 30.03 last 93.99) - create 4319 (time 1788560839.32 total 40.04 last 105.14) - create 5466 (time 1788560849.33 total 50.04 last 114.63) - create 6722 (time 1788560859.33 total 60.05 last 125.54) - create 7949 (time 1788560869.34 total 70.06 last 122.58) - create 9188 (time 1788560879.35 total 80.06 last 123.86) total: 10000 create in 86.83 seconds: 115.17 ops/second PASS 37 (99s) == sanityn test 39a: file mtime does not change after rename ========================================================== 18:28:14 (1788560894) repeat after cancel_lru_locks PASS 39a (9s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 18:28:24 (1788560904) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 18:28:31 (1788560911) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 18:28:40 (1788560920) fail_loc=0x411 fail_loc=0 PASS 39d (9s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 18:28:52 (1788560932) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (21s) == sanityn test 40c: pdirops: link and others ======================================================================== 18:29:13 (1788560953) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 18:29:31 (1788560971) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 18:29:49 (1788560989) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 18:30:05 (1788561005) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (15s) == sanityn test 41b: pdirops: create vs create ======================================================================== 18:30:20 (1788561020) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 18:30:32 (1788561032) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 18:30:46 (1788561046) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 18:31:01 (1788561061) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 18:31:16 (1788561076) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 18:31:30 (1788561090) Conflict PASS 41g (18s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 18:31:48 (1788561108) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (16s) == sanityn test 41i: reint_open: create vs create ======== 18:32:04 (1788561124) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1108s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 18:50:32 (1788562232) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 18:50:46 (1788562246) open: File exists Conflict PASS 42b (14s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 18:51:00 (1788562260) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (16s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 18:51:16 (1788562276) Conflict PASS 42d (18s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 18:51:35 (1788562295) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (16s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 18:51:51 (1788562311) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 18:52:05 (1788562325) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 18:52:20 (1788562340) Conflict PASS 42h (16s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 18:52:36 (1788562356) PASS 43a (80s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 18:53:56 (1788562436) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 18:54:10 (1788562450) Conflict PASS 43c (13s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 18:54:23 (1788562463) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (14s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 18:54:37 (1788562477) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 18:54:52 (1788562492) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 18:55:06 (1788562506) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (17s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 18:55:23 (1788562523) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 18:55:38 (1788562538) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 18:55:41 (1788562541) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (136s) == sanityn test 43k: unlink vs create ==================== 18:57:57 (1788562677) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1402s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 19:21:19 (1788564079) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 4 19:21:30 EDT 2026 PASS 44a (17s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 19:21:37 (1788564097) open: File exists Conflict PASS 44b (15s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 19:21:51 (1788564111) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 19:22:06 (1788564126) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 19:22:20 (1788564140) Conflict PASS 44e (16s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 19:22:37 (1788564157) Conflict PASS 44f (16s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 19:22:53 (1788564173) Conflict PASS 44g (16s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 19:23:09 (1788564189) Conflict PASS 44h (19s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 19:23:28 (1788564208) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 19:23:32 (1788564212) PASS 45a (123s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 19:25:35 (1788564335) Conflict PASS 45b (14s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 19:25:49 (1788564349) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 19:26:02 (1788564362) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (15s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 19:26:17 (1788564377) Conflict PASS 45e (19s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 19:26:36 (1788564396) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 19:26:50 (1788564410) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (13s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 19:27:03 (1788564423) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 19:27:17 (1788564437) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 19:27:21 (1788564441) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1272s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 19:48:34 (1788565714) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 19:48:47 (1788565727) open: File exists Conflict PASS 46b (16s) == sanityn test 46c: pdirops: link vs link ======================================================================== 19:49:03 (1788565743) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (15s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 19:49:18 (1788565758) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 19:49:31 (1788565771) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 19:49:46 (1788565786) Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 19:49:59 (1788565799) Conflict PASS 46g (16s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 19:50:15 (1788565815) Conflict PASS 46h (19s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 19:50:34 (1788565834) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (5s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 19:50:39 (1788565839) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (5s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 19:50:44 (1788565844) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (5s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 19:50:49 (1788565849) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (5s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 19:50:55 (1788565855) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (5s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 19:51:00 (1788565860) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 19:51:04 (1788565864) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 19:51:08 (1788565868) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 19:51:12 (1788565872) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.183842 s, 55.7 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (14s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 19:51:27 (1788565887) 0+1 records in 0+1 records out 158 bytes copied, 0.0323902 s, 4.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218645) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 19:51:37 (1788565897) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0150623 s, 68.0 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232677 s, 4.5 MB/s 3145728 PASS 51b (22s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 19:51:59 (1788565919) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0321475 s, 31.9 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 19:52:12 (1788565932) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601104 s, 17.4 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 19:52:21 (1788565941) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 19:52:31 (1788565951) llite.lustre-ffff98f48547f000.xattr_cache=1 llite.lustre-ffff98f4902c5000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:52:34.000000000 -0400 Modify: 2026-09-04 19:52:35.000000000 -0400 Change: 2026-09-04 19:52:35.000000000 -0400 Birth: 2026-09-04 19:52:35.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:52:42.000000000 -0400 Modify: 2026-09-04 19:52:43.000000000 -0400 Change: 2026-09-04 19:52:43.000000000 -0400 Birth: 2026-09-04 19:52:42.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:52:48.000000000 -0400 Modify: 2026-09-04 19:52:48.000000000 -0400 Change: 2026-09-04 19:52:48.000000000 -0400 Birth: 2026-09-04 19:52:49.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:52:48.000000000 -0400 Modify: 2026-09-04 19:52:50.000000000 -0400 Change: 2026-09-04 19:52:50.000000000 -0400 Birth: 2026-09-04 19:52:49.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:52:55.000000000 -0400 Modify: 2026-09-04 19:52:57.000000000 -0400 Change: 2026-09-04 19:52:57.000000000 -0400 Birth: 2026-09-04 19:52:56.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 19:52:55.000000000 -0400 Modify: 2026-09-04 19:52:57.000000000 -0400 Change: 2026-09-04 19:52:57.000000000 -0400 Birth: 2026-09-04 19:52:56.000000000 -0400 PASS 54 (38s) == sanityn test 55a: rename vs unlink target dir ========= 19:53:09 (1788565989) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (15s) == sanityn test 55b: rename vs unlink source dir ========= 19:53:24 (1788566004) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (16s) == sanityn test 55c: rename vs unlink orphan target dir == 19:53:40 (1788566020) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (22s) == sanityn test 55d: rename file vs link ================= 19:54:02 (1788566042) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 19:54:17 (1788566057) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 19:54:20 (1788566060) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (23s) == sanityn test 55g: rename: race with trylock =========== 19:54:43 (1788566083) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (26s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 19:55:09 (1788566109) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788566111 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.480623 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (140s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 19:57:29 (1788566249) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788566256 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.946312 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (147s) == sanityn test 60: Verify data_version behaviour ======== 19:59:56 (1788566396) PASS 60 (12s) == sanityn test 70a: cd directory && rm directory ======== 20:00:08 (1788566408) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 20:00:18 (1788566418) PASS 70b (13s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 20:00:31 (1788566431) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0154544 s, 4.2 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 20:00:35 (1788566435) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0442278 s, 1.5 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (6s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 20:00:41 (1788566441) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 20:00:46 (1788566446) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0145275 s, 4.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 20:00:51 (1788566451) llite.lustre-ffff98f48547f000.xattr_cache=1 llite.lustre-ffff98f4902c5000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (11s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 20:01:02 (1788566462) llite.lustre-ffff98f48547f000.xattr_cache=1 llite.lustre-ffff98f4902c5000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (11s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 20:01:12 (1788566472) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0215367 s, 47.5 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0259217 s, 39.5 kB/s 231480: get lock2 231482: get lock1 231482: done 231482: get lock2 231480: done 231480: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231480: cannot get lock1: Resource deadlock avoided 231482: done 231482: exit rc=0 231480: done 231480: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 20:01:24 (1788566484) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0374492 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033397 s, 31.4 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232064 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (14s) == sanityn test 76: Verify MDT open_files listing ======== 20:01:38 (1788566498) opening files: ........ closing files: ........ PASS 76 (85s) == sanityn test 77a: check FIFO NRS policy =============== 20:03:03 (1788566583) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.199165 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198647 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409076 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375795 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916683 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126697 s, 8.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117133 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166613 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109392 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320058 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550448 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119301 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978524 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136259 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194327 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211792 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360554 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387161 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984113 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603159 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652511 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978909 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086517 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155496 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682252 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00638831 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658408 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591251 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00706544 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911125 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00928784 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0032499 s, 323 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155818 s, 67.3 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (13s) == sanityn test 77b: check CRR-N NRS policy ============== 20:03:16 (1788566596) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.207372 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191182 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305498 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492606 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05382 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238397 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349204 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206906 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363578 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352953 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657243 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126114 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739966 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807902 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880404 s, 11.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102687 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195164 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029988 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561271 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341687 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787336 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022981 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411516 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496027 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274966 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449671 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00812633 s, 129 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370199 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00807379 s, 130 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335416 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597249 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00677421 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597605 s, 17.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.261333 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330626 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398008 s, 26.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157554 s, 6.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130096 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937637 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102444 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134553 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170213 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132436 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217839 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13991 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184368 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224876 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232791 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192408 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259376 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256333 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127194 s, 82.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895814 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951962 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140574 s, 74.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105596 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743621 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129713 s, 80.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662732 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191602 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728718 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566517 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124906 s, 83.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480127 s, 21.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116054 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394076 s, 26.6 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (13s) == sanityn test 77c: check ORR NRS policy ================ 20:03:30 (1788566610) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.271128 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261656 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672334 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777524 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150954 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152277 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119974 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195297 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11278 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179627 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114079 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16576 s, 6.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135269 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212289 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180498 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134459 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206291 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655369 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484978 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056493 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061187 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05546 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902603 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665291 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728491 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472916 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840869 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732891 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799793 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567256 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738441 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850384 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754079 s, 13.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.250716 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666446 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902426 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118218 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507424 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980391 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419066 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165971 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161269 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173087 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1358 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127068 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156744 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196448 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215209 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332865 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25309 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568638 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817944 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664217 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744496 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080167 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884715 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090695 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885599 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917233 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1224 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930164 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781631 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972288 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983818 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108742 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129707 s, 8.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.432036 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434001 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433399 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167305 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17493 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130849 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162247 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161444 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163067 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183971 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198823 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196234 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146918 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193932 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208606 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176813 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209034 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633431 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595332 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844857 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779447 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768968 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971429 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468538 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981385 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079156 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717497 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684039 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867211 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732813 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819856 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982634 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129921 s, 8.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (15s) == sanityn test 77d: check TRR nrs policy ================ 20:03:45 (1788566625) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.306124 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386689 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983611 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909028 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202375 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131538 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129649 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122405 s, 8.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470552 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506195 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137711 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950202 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661921 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169636 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117253 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169721 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119678 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716519 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522363 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341733 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414175 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447976 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341614 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355143 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831247 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879095 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532787 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553568 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110074 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140423 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00565433 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690552 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844605 s, 12.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.280381 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442635 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406035 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114758 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123484 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134115 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111035 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135117 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126436 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186609 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129249 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120173 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184151 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117752 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139312 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148608 s, 7.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211731 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509301 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00945215 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128759 s, 81.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151243 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123926 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127512 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302563 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273366 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130354 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712161 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677401 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100813 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11448 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100708 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105141 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157851 s, 6.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.359013 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395821 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682579 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773408 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150601 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161968 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149397 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227918 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177732 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199693 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152513 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133337 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171505 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237368 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139171 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16657 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20813 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106819 s, 9.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040886 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368661 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723241 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756175 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115371 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134523 s, 77.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126926 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085974 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798938 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832237 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705136 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912846 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838514 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826208 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942085 s, 11.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (17s) == sanityn test 77e: check TBF NID nrs policy ============ 20:04:02 (1788566642) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.57@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.312935 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273521 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222405 s, 47.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454629 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524162 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104526 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981869 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124075 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118148 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127847 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152979 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138999 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176363 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133079 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142961 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178959 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165251 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114069 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761096 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970331 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919301 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883894 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10432 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775689 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113015 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122569 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084923 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906485 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102782 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12041 s, 8.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764114 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118316 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939499 s, 11.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.430744 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400592 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617608 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775771 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118151 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839633 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711516 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154982 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146374 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239341 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783062 s, 13.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116759 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388024 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225097 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333655 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303107 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212958 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511291 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489259 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676229 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010536 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279332 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04621 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691737 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00695212 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104702 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109087 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517027 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346806 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00842174 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00640644 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109865 s, 95.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423593 s, 24.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.381009 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531223 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361118 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075161 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11817 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106563 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075772 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13848 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138686 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118567 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163979 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22151 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180808 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216787 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180417 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228691 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260519 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169604 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242775 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125174 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122711 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515481 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435019 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366091 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660914 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151565 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743942 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778961 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846681 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790107 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16548 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16393 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900443 s, 11.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.211174 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393569 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785952 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802845 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176765 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182167 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151974 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150345 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215341 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227121 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239929 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270333 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209543 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303259 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236401 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207678 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291188 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0050209 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576374 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585041 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625962 s, 16.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793701 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107258 s, 97.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145948 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426906 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100881 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00798969 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148412 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130881 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826185 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110508 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336862 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104456 s, 10.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (26s) == sanityn test 77f: check TBF JobID nrs policy ========== 20:04:28 (1788566668) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.23848 s, 70.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496769 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046625 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118035 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966109 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133132 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140199 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129351 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12683 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167748 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173483 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184962 s, 5.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189287 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189639 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212349 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184186 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220398 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832346 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353626 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420266 s, 25.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754993 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457605 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477169 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653643 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531975 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538205 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618491 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107581 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117486 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119402 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765533 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676547 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889878 s, 11.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.441437 s, 38.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576807 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105498 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109074 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133151 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256344 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220619 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697231 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264089 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202973 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101013 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175136 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118328 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434777 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199244 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421081 s, 2.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.490798 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104085 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155024 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975459 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118332 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138659 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156988 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140559 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158333 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171906 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183407 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15244 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163664 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195933 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198652 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180456 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206077 s, 5.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.310738 s, 54.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029897 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157328 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00929829 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027371 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733292 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041077 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407009 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696887 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995824 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921155 s, 11.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950545 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227797 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194165 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199298 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170712 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29599 s, 3.5 MB/s execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578785 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517541 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00761046 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00240767 s, 436 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171627 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102355 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149076 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157287 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00728518 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096306 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829578 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841278 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158387 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643213 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560627 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672704 s, 15.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.561887 s, 29.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197603 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016992 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620293 s, 16.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109378 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116407 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677706 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142428 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10875 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128636 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00703748 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191287 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237501 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063069 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306367 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505941 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152639 s, 6.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552015 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965094 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960482 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932086 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525605 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05948 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109101 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844577 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673618 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114547 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828293 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833729 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703956 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125748 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120824 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109253 s, 9.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (29s) == sanityn test 77g: Change TBF type directly ============ 20:04:57 (1788566697) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (14s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 20:05:12 (1788566712) oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 PASS 77h (14s) == sanityn test 77i: Change rank of TBF rule ============= 20:05:27 (1788566727) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (25s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 20:05:52 (1788566752) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.534931 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374886 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306877 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123339 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513233 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132185 s, 79.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889824 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562215 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714131 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175541 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189342 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126622 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169966 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125717 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134779 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117345 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15727 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253563 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438457 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454899 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544714 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486471 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829987 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533481 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610432 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956705 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830558 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696013 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727371 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882668 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916782 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095488 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111509 s, 9.4 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2798 s, 10.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7175 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (71s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 20:07:03 (1788566823) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.34495 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840186 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765484 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840338 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721929 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154837 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763941 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165593 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131246 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190479 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149765 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198329 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224181 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189658 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203165 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158713 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271461 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535845 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535575 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734792 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772095 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088376 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883896 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875498 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944916 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638837 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088831 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103461 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838209 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094034 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836642 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829444 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970825 s, 10.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1317 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9586 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28664 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735309 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934454 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938978 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136433 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217861 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160996 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12625 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198146 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282479 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237724 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278037 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154063 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344393 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171388 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33934 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35476 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116331 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140048 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121296 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131647 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978955 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101148 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091299 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195869 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994108 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161415 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107604 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115574 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118556 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234155 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164532 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159361 s, 6.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3692 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9093 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (162s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 20:09:45 (1788566985) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28765 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870056 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111713 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104809 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113896 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158116 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165496 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158019 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164682 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181871 s, 5.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200236 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182041 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19285 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231056 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231342 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295276 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340836 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742448 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710474 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834818 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572258 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879127 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963458 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961489 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103271 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101976 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968093 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101939 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116997 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116752 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123993 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122934 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11698 s, 9.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.4274 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.851 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.35322 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767072 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357419 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907866 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375918 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118835 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108174 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128821 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126912 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17623 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251777 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197002 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26497 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266135 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207777 s, 5.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234242 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236633 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368265 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584435 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232973 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213373 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478796 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507339 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601333 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081399 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543867 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569329 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961964 s, 10.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966752 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716374 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861863 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10658 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849761 s, 12.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9667 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.815 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (160s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 20:12:25 (1788567145) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.46499 s, 36.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630894 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554182 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109213 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147535 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616517 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165408 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703081 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180275 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016406 s, 63.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181507 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242558 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176086 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193142 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141521 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187944 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29372 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980918 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781416 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359271 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736543 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105016 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787199 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788636 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710879 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070475 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892533 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926117 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10326 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770991 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774456 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719463 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105486 s, 9.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.78101 s, 13.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98729 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.57@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.57@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.412291 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834714 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100403 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915959 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108505 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946554 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2251 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215872 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967073 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149639 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242319 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128636 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18232 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226504 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251127 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18059 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162787 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326974 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490487 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381264 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772638 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547621 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427522 s, 245 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569195 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064477 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059222 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597101 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716075 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739381 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106322 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831154 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730582 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925081 s, 11.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7456 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95795 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.57@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.386217 s, 43.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514853 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564149 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160373 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163258 s, 6.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141134 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152408 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13872 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19154 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134626 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145605 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192178 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20356 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189281 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225556 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166064 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240864 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147772 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140585 s, 74.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143345 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231408 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246693 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00935168 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00965266 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101425 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136454 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109598 s, 95.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933402 s, 11.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833645 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125791 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136208 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680868 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149648 s, 7.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.89452 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.26811 s, 19.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.59208 s, 28.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967405 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150199 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848902 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195147 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146324 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191942 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206865 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155655 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280199 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273897 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198648 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129856 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379743 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446386 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439402 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485975 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458913 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469519 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676016 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555621 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612621 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579755 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649824 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062057 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640043 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714287 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741757 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886052 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874267 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997089 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960578 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125305 s, 8.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4829 s, 9.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.071 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.51155 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91311 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.984032 s, 17.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697279 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805232 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274145 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208406 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22275 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236069 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276466 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184294 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242092 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298897 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237355 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379905 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30324 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347891 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331048 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331354 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155718 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110193 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164469 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175905 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224739 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174565 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122874 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177732 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136367 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161408 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138268 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247487 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159694 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194408 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153177 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157353 s, 6.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1612 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7291 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.399452 s, 42.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631657 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145685 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108861 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042113 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154706 s, 67.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128175 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633701 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866928 s, 12.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297104 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284871 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143462 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875252 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979204 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122662 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241594 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471889 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630436 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112031 s, 93.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107562 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140714 s, 74.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156976 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145527 s, 72.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975254 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388324 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283371 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142536 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213566 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110082 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150429 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292607 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660158 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778899 s, 13.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.59734 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95886 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.607412 s, 27.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334622 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528273 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125122 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12389 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153673 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149897 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176339 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190581 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205318 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180979 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197972 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20883 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183438 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217673 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25704 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282794 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513159 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513463 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102224 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105494 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109973 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695737 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672807 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673127 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905563 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905694 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934165 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827556 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955532 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919399 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137624 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151414 s, 6.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.357 s, 8.5 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96262 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.7438 s, 7.6 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0916 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (449s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 20:19:54 (1788567594) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.452824 s, 37.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028913 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298266 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032454 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428454 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105809 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713343 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751347 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169238 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214468 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203914 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215468 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186003 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169869 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232784 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243376 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274731 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0072433 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120809 s, 86.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129011 s, 81.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152633 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189251 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08245 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232809 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535254 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500494 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751659 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325715 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017563 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075385 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400215 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456045 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883731 s, 11.9 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0972 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.904 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (61s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 20:20:55 (1788567655) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.57@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.57@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.467022 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603448 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805377 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191804 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186104 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169533 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224478 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2128 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237442 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191288 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181958 s, 5.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277121 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187618 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25216 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262192 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233535 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245411 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351441 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558193 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455269 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702031 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691997 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101325 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668274 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746302 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807076 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083737 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855355 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811241 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921709 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139687 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101913 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974405 s, 10.8 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0062 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1044 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (58s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 20:21:53 (1788567713) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.57@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.607999 s, 27.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602126 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385137 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527805 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954918 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103787 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160168 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135997 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236411 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203755 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288442 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317786 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349751 s, 3.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269584 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35986 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284986 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338691 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff98f48547f000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118964 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979318 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100398 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911066 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10967 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119714 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136001 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111452 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139983 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104077 s, 10.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115505 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131332 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162271 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137908 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14362 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174628 s, 6.0 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1544 s, 9.4 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.69625 s, 18.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (53s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 20:22:46 (1788567766) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.582916 s, 28.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510401 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679212 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323675 s, 32.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453297 s, 23.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05947 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525592 s, 20.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496484 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116245 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149272 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668883 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10336 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925442 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196781 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174249 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184343 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205556 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574237 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476366 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640792 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058277 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742512 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904817 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693527 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706568 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804937 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873666 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762628 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829165 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889811 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897782 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100084 s, 10.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152579 s, 6.9 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.1256 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0783 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3434 s, 8.5 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0246 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (107s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 20:24:33 (1788567873) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.0418 s, 16.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404668 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700603 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619891 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994445 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142707 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105742 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950166 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688709 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746506 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133521 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195428 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175969 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276696 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282639 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222892 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282235 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792242 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598667 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810767 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699252 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651296 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747226 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107596 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862329 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966206 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925282 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993702 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103038 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877661 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934929 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908942 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10928 s, 9.6 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3811 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8729 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (79s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 20:25:52 (1788567952) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.63716 s, 26.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538348 s, 19.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700258 s, 15.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730003 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104752 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130232 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595387 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157396 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173143 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138408 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106588 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182996 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182117 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143023 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182941 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219234 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298116 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48547f000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074413 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741343 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13352 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970886 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934238 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138851 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106602 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102615 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106811 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119448 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18187 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184267 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130724 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135114 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136518 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154072 s, 6.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.67166 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0307 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.618716 s, 27.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566902 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425509 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105757 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253476 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174642 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182677 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251099 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205687 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255725 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343972 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267924 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237777 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322727 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303836 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443438 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425913 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455845 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00942988 s, 111 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111319 s, 94.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234625 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11565 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879813 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288507 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556842 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107557 s, 97.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135176 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542117 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420987 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719749 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684243 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582806 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696727 s, 15.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.588 s, 7.7 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0101 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.9034 s, 7.0 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9671 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (156s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 20:28:28 (1788568108) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:) Stopping client oleg457-client.virtnet /mnt/lustre opts: Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg457-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg457-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg457-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.57,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg457-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.57,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg457-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.57,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69586/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.57,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg457-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: kmemleak disabled oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: MODOPTS_LIBCFS= oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre Starting client oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Starting client oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98f483217800.idle_timeout=debug osc.lustre-OST0000-osc-ffff98f48368d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98f483217800.idle_timeout=debug osc.lustre-OST0001-osc-ffff98f48368d000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.567469 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113096 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134185 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176136 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162274 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181493 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224104 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27219 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211905 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350031 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420991 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210138 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301317 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314147 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.535105 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447655 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399428 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48368d000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173246 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199186 s, 5.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200839 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194967 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193853 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217889 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216033 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263241 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220213 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230027 s, 4.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238646 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241134 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243565 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244959 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313062 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277932 s, 3.8 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3772 s, 8.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.64499 s, 18.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.632931 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646057 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499877 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123388 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716963 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701404 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204596 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156073 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298059 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224202 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314299 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229094 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100158 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34656 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293961 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894484 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330166 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48368d000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff98f48368d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101847 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119213 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167432 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173506 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163164 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177477 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122255 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176244 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140007 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141358 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14871 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190954 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20046 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152569 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199052 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216926 s, 4.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98f48368d000.lock_unused_count=9 removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.64205 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98063 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (246s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 20:32:34 (1788568354) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (19s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 20:32:53 (1788568373) On MGS 192.168.204.157, active = nodemap.active=1 On MGS 192.168.204.157, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.551765 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960624 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117004 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235346 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223922 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268443 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233991 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332597 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306525 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351377 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325651 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350583 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372575 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345896 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339933 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326033 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455505 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03934 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650505 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420973 s, 24.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984639 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105527 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625716 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602033 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061936 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671834 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06561 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757238 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834383 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842835 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856625 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920367 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102061 s, 10.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.99637 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.36597 s, 19.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.157, TBF.id = On MGS 192.168.204.157, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.618699 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112954 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143669 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140363 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24244 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173734 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881707 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148528 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299145 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281498 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168399 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264525 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222017 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203065 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319364 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201207 s, 5.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98f48368d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278578 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05777 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488623 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846976 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596516 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557028 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10621 s, 9.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831104 s, 12.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855511 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574931 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925929 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794025 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123608 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100817 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147623 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111834 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117464 s, 8.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98f48368d000.lock_unused_count=3 removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.90732 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.77349 s, 37.8 MB/s Read runtime is 3 s, speed is 33.333333 IOPS read rate (33.333333) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 On MGS 192.168.204.157, active = nodemap.active=1 On MGS 192.168.204.157, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.686162 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10544 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181284 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167326 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797171 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120939 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299459 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767927 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308993 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352511 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411235 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212401 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389528 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290947 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23227 s, 4.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243043 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48368d000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff98f48368d000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224229 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201744 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189095 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18387 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199704 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228382 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21519 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20858 s, 5.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27828 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281677 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276191 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213346 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213832 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288024 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308306 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254123 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306261 s, 3.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff98f48368d000.lock_unused_count=6 removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.20736 s, 12.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0381 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.157, TBF.id = On MGS 192.168.204.157, active = nodemap.active=0 PASS 77kj (191s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 20:36:04 (1788568564) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (14s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 20:36:18 (1788568578) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg457-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 4.00518 s, 262 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg457-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.2117 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (61s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 20:37:20 (1788568640) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.841978 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117996 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160697 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170179 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27486 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361722 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315007 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302174 s, 3.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376424 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391838 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349892 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494904 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308176 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456042 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46195 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460766 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423148 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122558 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035416 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037945 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960391 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972596 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710399 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066797 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116511 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835046 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100549 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145641 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843222 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113215 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142108 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141925 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108725 s, 9.6 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.183 s, 8.6 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.76944 s, 18.2 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.870096 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178449 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161874 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171029 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225711 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418693 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455655 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35578 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32923 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401141 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367037 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359557 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353247 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.504613 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383219 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368079 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48368d000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff98f48368d000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.551762 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375712 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401314 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389805 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443063 s, 2.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390006 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407238 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402765 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440489 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474046 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff98f48368d000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45742 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.467932 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438477 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.525904 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476019 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468216 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.548287 s, 1.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.639 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.47396 s, 19.2 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (105s) == sanityn test 77o: Changing rank should not panic ====== 20:39:05 (1788568745) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (15s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 20:39:21 (1788568761) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (130s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 20:41:31 (1788568891) mds.MDS.mdt.nrs_policies=tbf oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg457-client: oleg457-server: ssh exited with exit code 36 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (44s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 20:42:15 (1788568935) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 272782 kill md_thread xargs PASS 77r (66s) == sanityn test 77s: Check TBF LRU shrinker ============== 20:43:21 (1788569001) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (72s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 20:44:33 (1788569073) oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg457-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg457-client: oleg457-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (11s) == sanityn test 79: xattr: intent error ================== 20:44:44 (1788569084) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935627 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 20:44:47.000000000 -0400 Modify: 2026-09-04 20:44:47.000000000 -0400 Change: 2026-09-04 20:44:47.000000000 -0400 Birth: 2026-09-04 20:44:47.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (20s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 20:45:05 (1788569105) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 20:45:09 (1788569109) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 20:45:13 (1788569113) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 20:45:17 (1788569117) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 20:45:21 (1788569121) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 20:45:26 (1788569126) - open/close 655 (time 1788569143.60 total 10.02 last 65.40) - open/close 1316 (time 1788569153.62 total 20.03 last 65.99) - open/close 1938 (time 1788569163.63 total 30.04 last 62.16) total: 2500 open/close in 38.11 seconds: 65.61 ops/second snapshot_time 1788569270.007273796 secs.nsecs start_time 1788569130.538013947 secs.nsecs elapsed_time 139.469259849 secs.nsecs open 2505 samples [usecs] 131 19196 4407260 13943845878 close 2504 samples [usecs] 168 9243 1015104 881779008 mknod 2501 samples [usecs] 614 18457 3845808 11035265254 unlink 2500 samples [usecs] 820 133477 6494398 92764695682 mkdir 2 samples [usecs] 1265 5157 6422 28194874 rmdir 2 samples [usecs] 19608 29206 48814 1237464100 rename 2500 samples [usecs] 538 47666 18573044 277627777548 getattr 3762 samples [usecs] 10 34491 293914 1517573838 parallel_rename_file 2500 samples [usecs] 538 47666 18573044 277627777548 crossdir_rename 2500 samples [usecs] 538 47666 18573044 277627777548 readpage 27 samples [usecs] 6641 98516 789830 35082643220 PASS 81d (149s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 20:47:55 (1788569275) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7646 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7646 PASS 82 (8s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 20:48:03 (1788569283) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (5s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 20:48:09 (1788569289) fail_loc=0x8000060b PASS 84 (16s) == sanityn test 85: Lustre API root cache race =========== 20:48:24 (1788569304) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788569306 Finishing test test1 at 1788569313 PASS 85 (16s) == sanityn test 90: open/create and unlink striped directory ========================================================== 20:48:41 (1788569321) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 20:48:46 (1788569326) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 20:48:50 (1788569330) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 20:48:55 (1788569335) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5753 0x1679 0x240000400 1 5714 0x1652 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5715 0x1653 0x280000400 0 5754 0x167a 0x240000400 rc1=0 and rc2=0 PASS 93 (20s) == sanityn test 94: signal vs CP callback race =========== 20:49:14 (1788569354) osc.lustre-OST0000-osc-ffff98f483217800.idle_timeout=0 osc.lustre-OST0000-osc-ffff98f48368d000.idle_timeout=0 osc.lustre-OST0001-osc-ffff98f483217800.idle_timeout=0 osc.lustre-OST0001-osc-ffff98f48368d000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.38811 s, 21.1 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.98539 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 289486 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0568665 s, 72.0 kB/s osc.lustre-OST0000-osc-ffff98f483217800.idle_timeout=debug osc.lustre-OST0000-osc-ffff98f48368d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98f483217800.idle_timeout=debug osc.lustre-OST0001-osc-ffff98f48368d000.idle_timeout=debug PASS 94 (19s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 20:49:33 (1788569373) llite.lustre-ffff98f483217800.fast_read=0 llite.lustre-ffff98f48368d000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110026 s, 19.1 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff98f483217800.fast_read=1 llite.lustre-ffff98f48368d000.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 20:49:54 (1788569394) llite.lustre-ffff98f483217800.fast_read=0 llite.lustre-ffff98f48368d000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00872408 s, 1.4 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff98f483217800.fast_read=1 llite.lustre-ffff98f48368d000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 20:50:09 (1788569409) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 20:50:12 (1788569412) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00559068 s, 733 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 20:50:20 (1788569420) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2142 s, 9.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 20:50:29 (1788569429) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.155434 s, 13.5 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (9s) == sanityn test 100e: DoM: read on open and file size ==== 20:50:38 (1788569438) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0226348 s, 530 kB/s PASS 100e (10s) == sanityn test 101a: Discard DoM data on unlink ========= 20:50:48 (1788569448) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.012761 s, 321 kB/s PASS 101a (10s) == sanityn test 101b: Discard DoM data on rename ========= 20:50:59 (1788569459) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0182703 s, 224 kB/s PASS 101b (10s) == sanityn test 101c: Discard DoM data on close-unlink === 20:51:09 (1788569469) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00906402 s, 452 kB/s PASS 101c (10s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 20:51:19 (1788569479) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 2c 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf2c:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (9s) == sanityn test 102b: open by handle of a dir inside an encrypted dir ========================================================== 20:51:29 (1788569489) 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) 192.168.204.157@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.204.157@tcp:/lustre /mnt/lustre Starting client: oleg457-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.204.157@tcp:/lustre /mnt/lustre2 file: /mnt/lustre/d102b.sanityn/encdir/sub fh_bytes: 32 fh_type: 151 fh_data: d3 0b 00 00 02 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd3:0x3:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! 192.168.204.157@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 PASS 102b (21s) == sanityn test 103: Test size correctness with lockahead ========================================================== 20:51:50 (1788569510) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788569513 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788569516 Finishing test test23 at 1788569517 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 20:52:05 (1788569525) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 20:52:09 (1788569529) fail_loc=0x416 PASS 105 (17s) == sanityn test 106a: Verify the btime via statx() ======= 20:52:26 (1788569546) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 20:52:30 (1788569550) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.035057 s, 29.2 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 20:52:32.000000000 -0400 Modify: 2026-09-04 20:52:32.000000000 -0400 Change: 2026-09-04 20:52:32.000000000 -0400 Birth: 2026-09-04 20:52:33.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238877265927 -rw-r--r-- 1 0 0 1788569553 1788569552 1788569552 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 20:52:32.000000000 -0400 Modify: 2026-09-04 20:52:32.000000000 -0400 Change: 2026-09-04 20:52:32.000000000 -0400 Birth: 2026-09-04 20:52:33.000000000 -0400 1788569552 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 20:52:39 (1788569559) PASS 106c (9s) == sanityn test 107a: Basic grouplock conflict =========== 20:52:48 (1788569568) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.278401 s, 37.7 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5722 0x165a 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 20:52:58 (1788569578) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.374309 s, 28.0 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5763 0x1683 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7646 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 20:53:12 (1788569592) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00873532 s, 1.1 MB/s llite.lustre-ffff98f485691000.hybrid_io=0 llite.lustre-ffff98f499b45000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.93366 s, 1.7 MB/s PASS 108a (14s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 20:53:27 (1788569607) 192.168.204.157@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre 192.168.204.157@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: kmemleak disabled oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: detected 4 online CPUs by sysfs oleg457-server: MODOPTS_LIBCFS= oleg457-server: Force libcfs to create 2 CPU partitions oleg457-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 PASS 109 (46s) == sanityn test 110: do not grant another lock on resend ========================================================== 20:54:13 (1788569653) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238910820355 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 20:54:15.000000000 -0400 Modify: 2026-09-04 20:54:15.000000000 -0400 Change: 2026-09-04 20:54:15.000000000 -0400 Birth: 2026-09-04 20:54:16.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg457-client.virtnet oleg457-client.virtnet PASS 110 (169s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 20:57:03 (1788569823) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 20:57:07 (1788569827) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 20:57:11 (1788569831) lustre-OST0000-osc-ffff98f483212800 active. lustre-OST0001-osc-ffff98f483212800 active. lustre-OST0000-osc-ffff98f483212800 active. lustre-OST0001-osc-ffff98f483212800 active. MGC192.168.204.157@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 20:57:19 (1788569839) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 20:57:23 (1788569843) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 20:57:26 (1788569846) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 20:57:30 (1788569850) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0173137 s, 1.2 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0940218 s, 218 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 20:57:39 (1788569859) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 20:57:50 (1788569870) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2515 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3358 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.289 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3151 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3001 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3324 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3486 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2345 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3542 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3474 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3677 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3948 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3619 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3986 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3828 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4182 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5947 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5774 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff98f483212800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff98f483212800.ost_server_uuid in FULL state after 0 sec PASS 200 (87s) cleanup: ====================================================== == sanityn test complete, duration 10927 sec ============= 20:59:20 (1788569960) === sanityn: start cleanup 20:59:23 (1788569963) === Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg457-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 21:04:22 (1788570262) ===