-----============= acceptance-small: sanityn ============----- Wed Sep 16 14:27:51 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg136-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 14:28:13 (1789583293) === Starting client oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2 Mount client oleg136-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2 Started clients oleg136-client.virtnet: 192.168.201.136@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg136-client.virtnet: executing check_config_client /mnt/lustre oleg136-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg136-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8aa484c95000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8aa492786800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8aa484c95000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8aa492786800.idle_timeout=debug disable quota as required oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 14:28:51 (1789583331) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303273 s, 34.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8447] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 14:28:53 (1789583333) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 14:29:03 (1789583343) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 14:29:10 (1789583350) -rw-r--r-- 1 root root 0 Sep 16 14:29 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 14:29:18 (1789583358) -rw-r--r-- 1 root root 0 Sep 16 14:29 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 14:29:24 (1789583364) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 14:29:31 (1789583371) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 14:29:38 (1789583378) total 1 -rwxrwxrwx 1 root root 0 Sep 16 14:29 f2a -rwxrwxrwx 1 root root 0 Sep 16 14:29 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 16 14:29 f2a -rwxrwxrwx 1 root root 0 Sep 16 14:29 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0169253 s, 30.3 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 14:29:46 (1789583386) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (8s) == sanityn test 2g: check blocks update on sync write ==== 14:29:54 (1789583394) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.522488 s, 4.0 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 14:30:02 (1789583402) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 14:30:08 (1789583408) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 14:30:17 (1789583417) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 14:30:24 (1789583424) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 14:30:32 (1789583432) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 14:30:38 (1789583438) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 14:30:45 (1789583445) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 14:30:53 (1789583453) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362933 s, 28.9 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0347346 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 14:30:59 (1789583459) 1+0 records in 1+0 records out 1 byte copied, 0.00594641 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00702122 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00750931 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00813434 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0115157 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00798007 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0132081 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00786643 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00837368 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00879487 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00628399 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00634889 s, 0.2 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 14:31:06 (1789583466) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0178576 s, 172 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0630014 s, 65.0 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00379745 s, 809 kB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 14:31:12 (1789583472) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 14:31:19 (1789583479) warning: '-runtime' deprecated, use '-t runtime' instead - link 338 (time 1789583491.45 total 10.02 last 33.74) - link 589 (time 1789583501.47 total 20.03 last 25.06) - link 891 (time 1789583511.47 total 30.04 last 30.19) - link 1247 (time 1789583521.49 total 40.05 last 35.55) - link 1587 (time 1789583531.51 total 50.07 last 33.92) - link 1901 (time 1789583541.52 total 60.08 last 31.37) - link 2284 (time 1789583551.53 total 70.09 last 38.27) - link 2652 (time 1789583561.55 total 80.12 last 36.70) - link 2991 (time 1789583571.58 total 90.14 last 33.81) total: 3347 link in 100.03 seconds: 33.46 ops/second using seed 1275008054 running for 100 seconds - stat 7071 (time 1789583493 ; total 11 ; last 11) - stat 10000 (time 1789583500 ; total 18 ; last 7) - stat 16406 (time 1789583511 ; total 29 ; last 11) - stat 20000 (time 1789583514 ; total 32 ; last 3) - stat 27672 (time 1789583525 ; total 43 ; last 11) - stat 30000 (time 1789583527 ; total 45 ; last 2) - stat 34919 (time 1789583538 ; total 56 ; last 11) - stat 39463 (time 1789583549 ; total 67 ; last 11) - stat 40000 (time 1789583549 ; total 67 ; last 0) - stat 43862 (time 1789583560 ; total 78 ; last 11) - stat 47750 (time 1789583571 ; total 89 ; last 11) - stat 50000 (time 1789583577 ; total 95 ; last 6) total: 51743 stats in 100 seconds: 517.429993 stats/second - unlinked 0 (time 1789583532 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3347) error: No such file or directory total: 3346 unlinks in 77 seconds: 43.454544 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22817) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22819) - No such process PASS 12 (137s) == sanityn test 13: test directory page revocation ======= 14:33:36 (1789583616) striped dir -i1 -c2 -H all_char /mnt/lustre/d13 f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 14:33:45 (1789583625) striped dir -i0 -c2 -H all_char /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 24028 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 14:33:52 (1789583632) striped dir -i0 -c2 -H crush /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 24619 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 14:34:00 (1789583640) striped dir -i0 -c2 -H all_char /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 25211 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 14:34:07 (1789583647) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 25805 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 14:34:14 (1789583654) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 26400 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 14:34:22 (1789583662) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7530496 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7530496KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (58s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 14:35:20 (1789583720) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.410983 s, 20.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.329002 s, 25.5 MB/s Chance of close/open is 1 in 50 Seed set to 3725 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1789583725.886002 write 0x7de716 thru 0x7e45b2 (0x5e9d bytes) 000002[0] 1789583725.917597 write 0x894dde thru 0x8a0a9b (0xbcbe bytes) 000003[0] 1789583725.934140 punch from 0x266a0d to 0x26b7b8 (0x4dac bytes) 000004[1] 1789583725.940610 mapread 0x23f469 thru 0x24d51d (0xe0b5 bytes) truncating to largest ever: 0x981e87 000006[1] 1789583726.013458 trunc from 0x8a0a9c to 0x981e86 (0xe13eb bytes) 000008[1] 1789583726.082729 read 0x34c5d3 thru 0x350917 (0x4345 bytes) 000009[0] 1789583726.091735 trunc from 0x981e87 to 0x19cb1b (0x7e536b bytes) 000011[0] 1789583726.157431 write 0x56af72 thru 0x570c4e (0x5cdd bytes) 000012[1] 1789583726.159953 read 0x210093 thru 0x2125e6 (0x2554 bytes) 000013[1] 1789583726.226342 punch from 0x4c1dd3 to 0x4c4eed (0x311b bytes) 000015[1] 1789583726.270386 write 0x9efd47 thru 0x9fbaec (0xbda6 bytes) 000016[0] 1789583726.289725 write 0x6f6774 thru 0x6fe984 (0x8211 bytes) 000017[0] 1789583726.320458 mapwrite 0x7ff6ee thru 0x80ad90 (0xb6a3 bytes) 000018[1] 1789583726.549832 punch from 0x350b81 to 0x35c44c (0xb8cc bytes) 000019[1] 1789583726.583701 punch from 0x8d68df to 0x8dec25 (0x8347 bytes) 000020[1] 1789583726.605081 write 0x7f67c8 thru 0x7fe550 (0x7d89 bytes) 000021[0] 1789583726.714553 mapread 0x4f72d2 thru 0x4f817b (0xeaa bytes) 000024[1] 1789583726.779023 mapwrite 0x47b3a6 thru 0x480abf (0x571a bytes) 000025[0] 1789583726.909478 punch from 0x4a2d1b to 0x4a2e4b (0x131 bytes) 000026[1] 1789583726.935537 trunc from 0x9fbaed to 0x2de02b (0x71dac1 bytes) 000029[1] 1789583727.028284 write 0x357eb2 thru 0x363e94 (0xbfe3 bytes) 000031[0] 1789583727.076889 mapwrite 0x998698 thru 0x999696 (0xfff bytes) 000032[0] 1789583727.158599 punch from 0x739e72 to 0x743489 (0x9618 bytes) 000033[1] 1789583727.188750 read 0x330d25 thru 0x339e64 (0x9140 bytes) 000034[1] 1789583727.215592 write 0x100ebb thru 0x103bd2 (0x2d18 bytes) 000037[0] 1789583727.288651 punch from 0x61dd3c to 0x6207be (0x2a83 bytes) 000038[0] 1789583727.312889 punch from 0x6f487c to 0x703be3 (0xf368 bytes) 000039[1] 1789583727.332698 read 0x284e8d thru 0x28c300 (0x7474 bytes) 000045[1] 1789583727.422329 trunc from 0x999697 to 0x0459f0 (0x953ca6 bytes) 000047[1] 1789583727.486411 punch from 0x022197 to 0x02656d (0x43d7 bytes) 000049[1] 1789583727.508146 write 0x7689e4 thru 0x76d2f4 (0x4911 bytes) 000050[0] 1789583727.513028 mapread 0x4e82da thru 0x4eb2e1 (0x3008 bytes) 000052[1] 1789583727.577044 mapwrite 0x686d52 thru 0x68bf42 (0x51f1 bytes) 000053[0] 1789583727.683420 mapread 0x3efcf5 thru 0x3f443b (0x4747 bytes) 000055[0] 1789583727.791848 mapwrite 0x5db9a0 thru 0x5eaabb (0xf11c bytes) 000059[1] 1789583727.950486 mapread 0x4250c3 thru 0x4254d8 (0x416 bytes) 000061[0] 1789583727.974185 write 0x8f0720 thru 0x8f7be7 (0x74c8 bytes) 000063[1] 1789583728.021475 read 0x55c798 thru 0x5696cf (0xcf38 bytes) 000064[0] 1789583728.032045 read 0x77dff2 thru 0x780dbb (0x2dca bytes) 000067[1] 1789583728.088463 write 0x664bac thru 0x66f549 (0xa99e bytes) 000073[1] 1789583728.193725 trunc from 0x8f7be8 to 0x50e0e9 (0x3e9afe bytes) 000074[1] 1789583728.239644 mapread 0x01abe1 thru 0x01c951 (0x1d71 bytes) 000076[0] 1789583728.257072 read 0x3c4d55 thru 0x3cb36f (0x661b bytes) 000077[1] 1789583728.293099 mapread 0x1f5839 thru 0x1f592f (0xf7 bytes) 000081[1] 1789583728.315093 punch from 0x2b8cd7 to 0x2bd787 (0x4ab1 bytes) 000082[0] 1789583728.354007 read 0x363901 thru 0x36725f (0x395f bytes) 000083[1] 1789583728.375286 punch from 0x02b450 to 0x030b45 (0x56f6 bytes) 000084[0] 1789583728.418159 write 0x98c352 thru 0x98e513 (0x21c2 bytes) 000086[0] 1789583728.458741 write 0x84dadf thru 0x858195 (0xa6b7 bytes) 000087[0] 1789583728.485146 write 0x378ca1 thru 0x37f6d6 (0x6a36 bytes) 000088[1] 1789583728.505717 mapwrite 0x7ebc81 thru 0x7f4b5e (0x8ede bytes) 000089[0] 1789583728.731120 punch from 0x38c957 to 0x398ab6 (0xc160 bytes) 000090[1] 1789583728.751949 mapread 0x82b6a2 thru 0x82c51e (0xe7d bytes) 000091[0] 1789583728.792028 write 0x09eca5 thru 0x0a421f (0x557b bytes) 000098[1] 1789583728.926720 mapread 0x6a8a48 thru 0x6ad2dd (0x4896 bytes) 000099[0] 1789583728.968944 mapwrite 0x37c260 thru 0x388feb (0xcd8c bytes) 000102[0] 1789583729.105674 trunc from 0x98e514 to 0x7e2854 (0x1abcbf bytes) 000103[0] 1789583729.153688 mapwrite 0x907da0 thru 0x90bfaf (0x4210 bytes) 000105[1] 1789583729.254481 trunc from 0x90bfb0 to 0x6b66dc (0x2558d3 bytes) 000108[0] 1789583729.298286 trunc from 0x6b66dd to 0x9802ff (0x2c9c23 bytes) 000110[1] 1789583729.366642 punch from 0x97f2db to 0x9802ff (0x1025 bytes) 000112[0] 1789583729.411455 mapwrite 0x983ced thru 0x992108 (0xe41c bytes) 000114[1] 1789583729.520592 read 0x7d3222 thru 0x7da6ac (0x748b bytes) 000115[1] 1789583729.555514 mapwrite 0x273e1f thru 0x27824e (0x4430 bytes) 000117[1] 1789583729.704935 punch from 0x259584 to 0x265f2e (0xc9ab bytes) 000118[1] 1789583729.710267 trunc from 0x992109 to 0x921815 (0x708f3 bytes) 000119[0] 1789583729.769946 trunc from 0x921816 to 0x5ece2b (0x3349ea bytes) 000120[1] 1789583729.856416 mapread 0x3977a6 thru 0x3a669c (0xeef7 bytes) 000121[0] 1789583729.993727 mapwrite 0x676407 thru 0x67890f (0x2509 bytes) 000124[1] 1789583730.282262 trunc from 0x678910 to 0x7136de (0x9adcf bytes) 000125[0] 1789583730.375713 read 0x0128a4 thru 0x021a50 (0xf1ad bytes) 000127[0] 1789583730.444259 write 0x72ebb0 thru 0x737b18 (0x8f69 bytes) 000132[1] 1789583730.540682 read 0x55ae52 thru 0x569830 (0xe9df bytes) 000133[1] 1789583730.586089 trunc from 0x737b19 to 0x3f57b9 (0x34235f bytes) 000135[1] 1789583730.704722 mapread 0x383288 thru 0x389df5 (0x6b6e bytes) 000136[1] 1789583730.884662 trunc from 0x3f57ba to 0x6cbe09 (0x2d6650 bytes) 000137[0] 1789583730.965852 trunc from 0x6cbe0a to 0x103ab4 (0x5c8355 bytes) 000139[1] 1789583731.040428 mapread 0x062c30 thru 0x06a0e4 (0x74b5 bytes) 000141[0] 1789583731.149388 trunc from 0x103ab5 to 0x1dd0a3 (0xd95ef bytes) 000142[0] 1789583731.292729 trunc from 0x1dd0a4 to 0x84dc9e (0x670bfb bytes) 000143[1] 1789583731.382747 punch from 0x76ac06 to 0x771117 (0x6512 bytes) 000144[1] 1789583731.458016 trunc from 0x84dc9f to 0x702a17 (0x14b287 bytes) 000145[0] 1789583731.586150 trunc from 0x702a18 to 0x4c197c (0x24109b bytes) 000146[1] 1789583731.703551 mapwrite 0x6c930a thru 0x6d7d02 (0xe9f9 bytes) 000147[1] 1789583731.916662 write 0x3f47a1 thru 0x3fe385 (0x9be5 bytes) 000148[0] 1789583731.997722 mapwrite 0x35234c thru 0x3597e1 (0x7496 bytes) 000149[1] 1789583732.240719 read 0x55d4ba thru 0x55eaac (0x15f3 bytes) 000151[0] 1789583732.285526 mapread 0x38a699 thru 0x39a2a7 (0xfc0f bytes) 000155[1] 1789583732.447400 punch from 0x49f4be to 0x4ac39e (0xcee1 bytes) 000156[1] 1789583732.479273 write 0x0d4b35 thru 0x0def8e (0xa45a bytes) 000158[0] 1789583732.547114 mapread 0x504394 thru 0x509199 (0x4e06 bytes) 000160[0] 1789583732.737278 mapread 0x2383b1 thru 0x24088b (0x84db bytes) 000162[0] 1789583732.873196 punch from 0x284fb7 to 0x28de9c (0x8ee6 bytes) 000163[0] 1789583732.911952 trunc from 0x6d7d03 to 0x970b0a (0x298e08 bytes) 000164[0] 1789583732.988119 trunc from 0x970b0b to 0x145bfa (0x82af10 bytes) 000165[1] 1789583733.052475 read 0x0308e9 thru 0x03543c (0x4b54 bytes) 000166[0] 1789583733.157068 write 0x5aeb33 thru 0x5b197a (0x2e48 bytes) 000167[1] 1789583733.200693 write 0x830f8b thru 0x83d1cd (0xc243 bytes) 000168[0] 1789583733.267313 punch from 0x5c4749 to 0x5d0f64 (0xc81c bytes) 000170[0] 1789583733.381215 trunc from 0x83d1ce to 0x6b638f (0x186e3e bytes) 000171[0] 1789583733.475458 trunc from 0x6b6390 to 0x129443 (0x58cf4c bytes) 000173[1] 1789583733.545729 punch from 0x0f3144 to 0x0f52c1 (0x217e bytes) 000174[1] 1789583733.591562 punch from 0x111f92 to 0x119f67 (0x7fd6 bytes) 000178[0] 1789583733.696596 read 0x0e198e thru 0x0e34bb (0x1b2e bytes) 000179[0] 1789583733.797232 mapread 0x025f69 thru 0x02de47 (0x7edf bytes) 000180[1] 1789583733.808616 trunc from 0x129444 to 0x4f4ad9 (0x3cb696 bytes) 000182[0] 1789583734.005399 trunc from 0x4f4ada to 0x11e264 (0x3d6875 bytes) 000183[1] 1789583734.118655 mapread 0x02f591 thru 0x0362f4 (0x6d64 bytes) 000184[0] 1789583734.338171 read 0x02cb95 thru 0x030da2 (0x420e bytes) 000188[1] 1789583734.495256 mapread 0x10e79f thru 0x116cfc (0x855e bytes) 000189[1] 1789583734.691413 mapwrite 0x37b85f thru 0x380af0 (0x5292 bytes) 000193[0] 1789583735.295090 mapread 0x12e25d thru 0x134ce8 (0x6a8c bytes) 000194[0] 1789583735.363106 mapwrite 0x232d29 thru 0x240acc (0xdda4 bytes) 000196[1] 1789583735.610540 trunc from 0x380af1 to 0x51aeeb (0x19a3fb bytes) 000197[1] 1789583735.651098 mapwrite 0x735bfe thru 0x74127e (0xb681 bytes) 000198[1] 1789583735.879138 mapread 0x550387 thru 0x555426 (0x50a0 bytes) 000199[0] 1789583735.946326 mapread 0x4d432c thru 0x4def04 (0xabd9 bytes) 000201[1] 1789583736.174080 punch from 0x4bc23b to 0x4c0694 (0x445a bytes) 000202[1] 1789583736.210398 mapread 0x6f6ba4 thru 0x6f7a3f (0xe9c bytes) 000203[1] 1789583736.226667 trunc from 0x74127f to 0x015b9f (0x72b6df bytes) 000204[1] 1789583736.266781 mapread 0x012f61 thru 0x015b9f (0x2c3f bytes) 000206[0] 1789583736.335116 trunc from 0x015ba0 to 0x0b87f4 (0xa2c55 bytes) 000208[0] 1789583736.418147 read 0x061fc2 thru 0x069d1d (0x7d5c bytes) 000209[1] 1789583736.433927 punch from 0x03bf10 to 0x0489b5 (0xcaa6 bytes) 000210[0] 1789583736.458514 write 0x8fcc6f thru 0x9046f8 (0x7a8a bytes) 000212[1] 1789583736.493331 write 0x797d45 thru 0x79afc8 (0x3284 bytes) 000214[1] 1789583736.545583 write 0x943b0a thru 0x9489b9 (0x4eb0 bytes) 000215[0] 1789583736.605417 read 0x2158a3 thru 0x220005 (0xa763 bytes) 000217[1] 1789583736.646290 punch from 0x3cb30b to 0x3d6ad3 (0xb7c9 bytes) 000218[1] 1789583736.702068 mapread 0x70ad83 thru 0x712870 (0x7aee bytes) 000219[0] 1789583736.791054 trunc from 0x9489ba to 0x7de334 (0x16a685 bytes) 000221[1] 1789583736.959711 punch from 0x6180fe to 0x621b37 (0x9a3a bytes) 000223[1] 1789583737.056306 mapread 0x5cb2e7 thru 0x5d229b (0x6fb5 bytes) 000225[0] 1789583737.245829 punch from 0x5382dc to 0x53c5d6 (0x42fb bytes) 000226[0] 1789583737.314168 write 0x0c88fc thru 0x0d04fc (0x7c01 bytes) 000229[1] 1789583737.402691 punch from 0x1f822d to 0x201324 (0x90f8 bytes) 000233[0] 1789583737.526161 mapread 0x2e10ee thru 0x2e6dd4 (0x5ce7 bytes) 000234[0] 1789583737.582552 mapread 0x138c8c thru 0x146003 (0xd378 bytes) 000235[0] 1789583737.650714 write 0x334a84 thru 0x3425af (0xdb2c bytes) 000237[1] 1789583737.735406 mapwrite 0x5cf944 thru 0x5d701f (0x76dc bytes) 000238[1] 1789583737.915753 trunc from 0x7de335 to 0x435682 (0x3a8cb2 bytes) 000242[0] 1789583738.016537 mapwrite 0x695609 thru 0x69c181 (0x6b79 bytes) 000243[1] 1789583738.296947 mapread 0x3be8f8 thru 0x3c9556 (0xac5f bytes) 000244[1] 1789583738.487989 mapwrite 0x4fd87a thru 0x504c3d (0x73c4 bytes) 000245[0] 1789583738.685601 write 0x152607 thru 0x158264 (0x5c5e bytes) 000248[1] 1789583738.827330 mapread 0x18ec08 thru 0x19e0bb (0xf4b4 bytes) 000249[1] 1789583739.046251 mapwrite 0x3327e8 thru 0x33b92a (0x9143 bytes) 000250[0] 1789583739.334084 write 0x34467e thru 0x34eb43 (0xa4c6 bytes) 000252[0] 1789583739.449095 mapwrite 0x8715e6 thru 0x87b161 (0x9b7c bytes) 000253[1] 1789583739.743046 mapread 0x4084c5 thru 0x4135e2 (0xb11e bytes) 000254[0] 1789583739.911812 mapwrite 0x3b1c8e thru 0x3be107 (0xc47a bytes) 000255[0] 1789583740.125520 trunc from 0x87b162 to 0x387ba3 (0x4f35be bytes) 000256[1] 1789583740.213478 punch from 0x275b96 to 0x276c51 (0x10bc bytes) 000257[0] 1789583740.260196 write 0x9c4c3f thru 0x9d18a6 (0xcc68 bytes) 000259[0] 1789583740.271973 punch from 0x3fe312 to 0x40a679 (0xc368 bytes) 000260[0] 1789583740.318636 write 0x48031d thru 0x4806b8 (0x39c bytes) 000262[1] 1789583740.362365 mapwrite 0x79a4e3 thru 0x79d3ae (0x2ecc bytes) 000263[1] 1789583740.753490 write 0x3c2363 thru 0x3d04cf (0xe16d bytes) 000264[1] 1789583740.850255 read 0x26709e thru 0x26e8b4 (0x7817 bytes) 000266[0] 1789583740.934560 punch from 0x5dab5c to 0x5db197 (0x63c bytes) 000267[0] 1789583740.980880 mapwrite 0x87d9d7 thru 0x88341f (0x5a49 bytes) 000270[0] 1789583741.293745 mapwrite 0x759b51 thru 0x75a6a9 (0xb59 bytes) 000271[0] 1789583741.394970 mapwrite 0x077ecd thru 0x07eea4 (0x6fd8 bytes) 000276[0] 1789583741.723352 write 0x69f4a4 thru 0x6a08aa (0x1407 bytes) 000279[0] 1789583741.882964 mapread 0x18690e thru 0x18c66c (0x5d5f bytes) 000280[1] 1789583741.926014 trunc from 0x9d18a7 to 0x40d165 (0x5c4741 bytes) 000281[0] 1789583742.096720 mapwrite 0x149e98 thru 0x14b724 (0x188d bytes) 000282[1] 1789583742.336974 trunc from 0x40d166 to 0x6a6f1b (0x299db6 bytes) 000285[0] 1789583742.489327 mapread 0x12c27b thru 0x13a4eb (0xe271 bytes) 000287[1] 1789583742.602473 trunc from 0x6a6f1c to 0x74ea8a (0xa7b6f bytes) 000289[1] 1789583742.712594 read 0x663e93 thru 0x66a068 (0x61d6 bytes) 000290[0] 1789583742.722596 write 0x0c6170 thru 0x0d25a4 (0xc435 bytes) 000291[0] 1789583742.795883 mapread 0x72bcd6 thru 0x72dded (0x2118 bytes) 000293[1] 1789583742.898403 trunc from 0x74ea8b to 0x40decb (0x340bbf bytes) 000296[1] 1789583743.134081 write 0x6f7305 thru 0x706bb1 (0xf8ad bytes) 000297[0] 1789583743.147927 punch from 0x43d01a to 0x4490a6 (0xc08d bytes) 000299[0] 1789583743.236557 trunc from 0x706bb2 to 0x2bb0cb (0x44bae6 bytes) 000300[1] 1789583743.267294 read 0x131911 thru 0x133036 (0x1726 bytes) 000301[0] 1789583743.369457 write 0x3ec7c4 thru 0x3f2427 (0x5c64 bytes) 000302[0] 1789583743.446406 mapread 0x0cbb8f thru 0x0d0746 (0x4bb8 bytes) 000303[0] 1789583743.495046 write 0x8d924e thru 0x8d9ac2 (0x875 bytes) 000304[1] 1789583743.558457 punch from 0x5cdf79 to 0x5cf881 (0x1909 bytes) 000305[0] 1789583743.602831 write 0x1db1d0 thru 0x1ddcad (0x2ade bytes) 000306[1] 1789583743.640730 write 0x01a43d thru 0x01f083 (0x4c47 bytes) 000307[1] 1789583743.683443 trunc from 0x8d9ac3 to 0x7a8ccd (0x130df5 bytes) 000308[0] 1789583743.747358 trunc from 0x7a8cce to 0x8851e4 (0xdc517 bytes) 000310[1] 1789583743.812795 mapread 0x418abb thru 0x4282bf (0xf805 bytes) 000311[1] 1789583743.921810 write 0x952d40 thru 0x9578bf (0x4b80 bytes) 000315[1] 1789583743.957024 read 0x6f4035 thru 0x6f9548 (0x5514 bytes) 000317[0] 1789583744.006871 write 0x955e5b thru 0x961ed2 (0xc078 bytes) 000320[0] 1789583744.101597 mapwrite 0x4665f4 thru 0x46d108 (0x6b15 bytes) 000321[1] 1789583744.246987 mapwrite 0x5c8ab5 thru 0x5cfbe9 (0x7135 bytes) 000322[1] 1789583744.505708 punch from 0x3a48a0 to 0x3ac3b5 (0x7b16 bytes) 000323[0] 1789583744.549739 mapread 0x5cdb54 thru 0x5d30b9 (0x5566 bytes) 000324[1] 1789583744.665336 mapwrite 0x783ec4 thru 0x7923c6 (0xe503 bytes) 000325[0] 1789583744.879495 mapread 0x3b66d5 thru 0x3b837e (0x1caa bytes) 000326[0] 1789583745.023158 mapwrite 0x1b59b9 thru 0x1c19e1 (0xc029 bytes) 000328[0] 1789583745.141148 punch from 0x628c23 to 0x631213 (0x85f1 bytes) 000329[0] 1789583745.174577 punch from 0x295f33 to 0x29d709 (0x77d7 bytes) 000330[0] 1789583745.196916 mapwrite 0x1b0dcf thru 0x1b1a48 (0xc7a bytes) 000331[1] 1789583745.231525 punch from 0x6810ac to 0x68e435 (0xd38a bytes) 000332[0] 1789583745.256168 write 0x29df29 thru 0x2a2f19 (0x4ff1 bytes) 000333[0] 1789583745.275235 punch from 0x783447 to 0x790e37 (0xd9f1 bytes) 000336[0] 1789583745.326662 write 0x825dff thru 0x82b5f8 (0x57fa bytes) 000337[1] 1789583745.336298 punch from 0x0a7361 to 0x0aa7eb (0x348b bytes) 000338[1] 1789583745.388872 punch from 0x265b90 to 0x267290 (0x1701 bytes) 000340[1] 1789583745.416533 read 0x1e97c4 thru 0x1f6bc0 (0xd3fd bytes) 000343[0] 1789583745.480355 mapwrite 0x454de5 thru 0x46201d (0xd239 bytes) 000344[0] 1789583745.576032 mapread 0x2dbfc2 thru 0x2e7ac4 (0xbb03 bytes) 000346[1] 1789583745.654390 read 0x4e1b42 thru 0x4e2c5d (0x111c bytes) 000347[0] 1789583745.672242 mapread 0x682d66 thru 0x69270d (0xf9a8 bytes) 000350[0] 1789583745.701854 punch from 0x7735a2 to 0x77ac99 (0x76f8 bytes) 000352[1] 1789583745.739991 write 0x9e1b68 thru 0x9e7a13 (0x5eac bytes) 000353[1] 1789583745.761368 trunc from 0x9e7a14 to 0x2bdece (0x729b45 bytes) 000354[0] 1789583745.808755 mapwrite 0x1fd2a5 thru 0x1feeb3 (0x1c0f bytes) 000355[1] 1789583745.947898 trunc from 0x2bdecf to 0x0af95b (0x20e573 bytes) 000356[0] 1789583745.975014 write 0x4a0248 thru 0x4acbde (0xc997 bytes) 000357[0] 1789583745.997994 mapwrite 0x6768c7 thru 0x677be9 (0x1323 bytes) 000359[1] 1789583746.081410 write 0x1b13e8 thru 0x1c0631 (0xf24a bytes) 000360[1] 1789583746.147395 mapwrite 0x2f4783 thru 0x303d7e (0xf5fc bytes) 000361[0] 1789583746.235899 mapwrite 0x32a578 thru 0x331e91 (0x791a bytes) 000367[0] 1789583746.443396 write 0x883f10 thru 0x88e09f (0xa190 bytes) 000370[0] 1789583746.492656 mapwrite 0x305b48 thru 0x30883d (0x2cf6 bytes) 000371[0] 1789583746.549306 write 0x4a86bd thru 0x4b55af (0xcef3 bytes) 000372[1] 1789583746.572750 write 0x9f7d6b thru 0x9fffff (0x8295 bytes) 000373[1] 1789583746.602969 mapread 0x83c282 thru 0x842f8a (0x6d09 bytes) 000374[1] 1789583746.727318 write 0x9e4d98 thru 0x9e8fec (0x4255 bytes) 000376[1] 1789583746.821180 write 0x8aa85f thru 0x8b5962 (0xb104 bytes) 000377[1] 1789583746.899540 write 0x65c7b5 thru 0x669eb7 (0xd703 bytes) 000378[0] 1789583746.955877 mapwrite 0x080a4c thru 0x08200c (0x15c1 bytes) 000380[1] 1789583747.335069 read 0x0ef958 thru 0x0f7815 (0x7ebe bytes) 000381[0] 1789583747.385096 punch from 0x760557 to 0x764fff (0x4aa9 bytes) 000384[1] 1789583747.450008 read 0x9311bc thru 0x9397a9 (0x85ee bytes) 000385[1] 1789583747.489503 trunc from 0xa00000 to 0x1c08d2 (0x83f72d bytes) 000386[0] 1789583747.578950 write 0x9c21a0 thru 0x9c92e2 (0x7143 bytes) 000391[0] 1789583747.704350 mapwrite 0x9d5d77 thru 0x9e0e6c (0xb0f6 bytes) 000393[0] 1789583748.000068 punch from 0x7a0373 to 0x7aa7b1 (0xa43f bytes) 000395[1] 1789583748.008041 punch from 0x96347f to 0x97185b (0xe3dd bytes) 000396[1] 1789583748.065910 trunc from 0x9e0e6d to 0x21f23f (0x7c1c2d bytes) 000398[0] 1789583748.200835 trunc from 0x21f240 to 0x075cfc (0x1a9543 bytes) 000401[1] 1789583748.267051 mapread 0x073ec1 thru 0x075cfc (0x1e3c bytes) 000402[1] 1789583748.394131 read 0x05f639 thru 0x06e5cf (0xef97 bytes) 000406[1] 1789583748.584319 trunc from 0x075cfd to 0x15a9ce (0xe4cd2 bytes) 000408[0] 1789583748.716670 mapwrite 0x4ebadb thru 0x4ec5b9 (0xadf bytes) 000409[0] 1789583748.975683 mapwrite 0x980208 thru 0x982191 (0x1f8a bytes) 000410[1] 1789583749.081908 read 0x9623a7 thru 0x96d10d (0xad67 bytes) 000414[0] 1789583749.189264 punch from 0x3269f2 to 0x32ef6d (0x857c bytes) 000415[1] 1789583749.216361 trunc from 0x982192 to 0x7715e9 (0x210ba8 bytes) 000416[1] 1789583749.338279 write 0x02a4f9 thru 0x0368cd (0xc3d5 bytes) 000417[1] 1789583749.431985 read 0x5e1a7c thru 0x5e433c (0x28c1 bytes) 000418[1] 1789583749.525632 write 0x2b2637 thru 0x2b72f4 (0x4cbe bytes) 000420[0] 1789583749.609163 trunc from 0x7715ea to 0x5f2977 (0x17ec72 bytes) 000421[0] 1789583749.748292 trunc from 0x5f2978 to 0x0ddd0f (0x514c68 bytes) 000423[0] 1789583749.812149 read 0x068f00 thru 0x0756db (0xc7dc bytes) 000424[1] 1789583749.854435 trunc from 0x0ddd10 to 0x1f3da2 (0x116093 bytes) 000425[0] 1789583749.944167 read 0x1662f5 thru 0x174e9a (0xeba6 bytes) 000426[0] 1789583750.056981 punch from 0x04096d to 0x045118 (0x47ac bytes) 000427[1] 1789583750.135810 mapread 0x16c3e8 thru 0x172eec (0x6b05 bytes) 000428[0] 1789583750.220258 mapread 0x1d1fd6 thru 0x1e09e7 (0xea12 bytes) 000430[0] 1789583750.407906 write 0x305487 thru 0x305bc3 (0x73d bytes) 000431[1] 1789583750.467432 write 0x271c70 thru 0x27b80d (0x9b9e bytes) 000432[0] 1789583750.529290 trunc from 0x305bc4 to 0x23f63f (0xc6584 bytes) 000434[0] 1789583750.582020 mapwrite 0x4aef91 thru 0x4b9d88 (0xadf8 bytes) 000437[0] 1789583750.797727 mapread 0x02f187 thru 0x036745 (0x75bf bytes) 000438[0] 1789583750.888045 mapwrite 0x9b6d59 thru 0x9bf446 (0x86ee bytes) 000439[1] 1789583751.263865 mapwrite 0x58183a thru 0x58a5fc (0x8dc3 bytes) 000440[1] 1789583751.478080 trunc from 0x9bf447 to 0x109e2d (0x8b5619 bytes) 000441[0] 1789583751.542467 write 0x2daacb thru 0x2db258 (0x78e bytes) 000443[0] 1789583751.641444 mapread 0x0fbf3a thru 0x10232e (0x63f5 bytes) 000444[1] 1789583751.736512 read 0x1c2d45 thru 0x1cb497 (0x8753 bytes) 000445[0] 1789583751.826797 mapread 0x116d22 thru 0x118236 (0x1515 bytes) 000446[0] 1789583751.865376 mapwrite 0x3c9b52 thru 0x3c9e43 (0x2f2 bytes) 000448[0] 1789583752.240853 mapread 0x2ace2f thru 0x2ad205 (0x3d7 bytes) 000449[0] 1789583752.321901 mapread 0x3884a0 thru 0x38bfd1 (0x3b32 bytes) 000450[1] 1789583752.373745 trunc from 0x3c9e44 to 0x64b44d (0x28160a bytes) 000451[1] 1789583752.425397 mapread 0x0df6ed thru 0x0e7f8e (0x88a2 bytes) 000452[0] 1789583752.513347 mapread 0x270177 thru 0x278126 (0x7fb0 bytes) 000453[0] 1789583752.602422 mapwrite 0x3158b7 thru 0x31657a (0xcc4 bytes) 000454[1] 1789583752.693584 trunc from 0x64b44e to 0x88e87e (0x243431 bytes) 000457[0] 1789583752.793263 trunc from 0x88e87f to 0x267c17 (0x626c67 bytes) 000459[0] 1789583752.912284 mapread 0x0c34f3 thru 0x0c7f8d (0x4a9b bytes) 000460[0] 1789583753.030329 read 0x07daad thru 0x07f92a (0x1e7e bytes) 000461[0] 1789583753.093566 trunc from 0x267c18 to 0x960a6d (0x6f8e56 bytes) 000464[1] 1789583753.246400 write 0x290bee thru 0x29aad4 (0x9ee7 bytes) 000465[0] 1789583753.337660 write 0x185494 thru 0x18f4e0 (0xa04d bytes) 000466[0] 1789583753.482389 read 0x179f7d thru 0x17a42c (0x4b0 bytes) 000467[0] 1789583753.489953 mapread 0x16a871 thru 0x16dd94 (0x3524 bytes) 000468[1] 1789583753.585023 punch from 0x4fb44f to 0x4fce81 (0x1a33 bytes) 000470[1] 1789583753.634514 mapwrite 0x08246f thru 0x0903f1 (0xdf83 bytes) 000471[1] 1789583753.801678 mapread 0x2323c7 thru 0x233977 (0x15b1 bytes) 000472[0] 1789583753.813228 write 0x8fe2ff thru 0x906bc8 (0x88ca bytes) 000473[0] 1789583753.853339 read 0x8c3251 thru 0x8d020b (0xcfbb bytes) 000474[1] 1789583753.886621 punch from 0x00ddf0 to 0x013855 (0x5a66 bytes) 000475[1] 1789583753.924320 write 0x8c36a7 thru 0x8cbd7d (0x86d7 bytes) 000476[1] 1789583754.038405 write 0x4c1c7f thru 0x4c4de7 (0x3169 bytes) 000477[1] 1789583754.068405 punch from 0x7c5e72 to 0x7ca5ac (0x473b bytes) 000478[1] 1789583754.103464 mapread 0x17c300 thru 0x18a8aa (0xe5ab bytes) 000480[1] 1789583754.208287 read 0x3e062f thru 0x3efc3b (0xf60d bytes) 000482[1] 1789583754.233304 punch from 0x4ca551 to 0x4da165 (0xfc15 bytes) 000483[1] 1789583754.288637 mapread 0x8dcfa1 thru 0x8e38f1 (0x6951 bytes) 000484[1] 1789583754.353299 punch from 0x2e3cd1 to 0x2ef115 (0xb445 bytes) 000485[1] 1789583754.388034 mapwrite 0x297b8a thru 0x2a3ff5 (0xc46c bytes) 000488[1] 1789583754.545944 mapwrite 0x61128a thru 0x619d54 (0x8acb bytes) 000489[1] 1789583754.660910 read 0x174186 thru 0x17a915 (0x6790 bytes) 000493[1] 1789583754.718047 mapread 0x1a413f thru 0x1a66f4 (0x25b6 bytes) 000495[0] 1789583754.783938 read 0x40d40e thru 0x4100f5 (0x2ce8 bytes) 000497[0] 1789583754.852385 read 0x420f23 thru 0x427d49 (0x6e27 bytes) 000498[0] 1789583754.868562 write 0x7d931b thru 0x7deddc (0x5ac2 bytes) 000499[0] 1789583754.916726 write 0x1052c6 thru 0x10b106 (0x5e41 bytes) 000500[0] 1789583754.944616 trunc from 0x960a6e to 0x464b08 (0x4fbf65 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3755 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x51f17d 000002[1] 1789583755.752147 trunc from 00000000 to 0x51f17c (0x51f17d bytes) 000003[1] 1789583755.829959 read 0x3b5000 thru 0x3c3263 (0xe264 bytes) 000006[0] 1789583755.894203 punch from 0x0c7721 to 0x0c8c33 (0x1513 bytes) 000009[1] 1789583755.935656 punch from 0x2873d8 to 0x2960be (0xece7 bytes) 000010[0] 1789583755.969647 write 0x1e7000 thru 0x1e79c7 (0x9c8 bytes) 000012[1] 1789583756.040742 write 0x053000 thru 0x05d828 (0xa829 bytes) 000017[0] 1789583756.112610 read 0x00c000 thru 0x011387 (0x5388 bytes) 000021[1] 1789583756.181852 read 0x0d4000 thru 0x0e1498 (0xd499 bytes) 000024[1] 1789583756.212211 mapread 0x3ea000 thru 0x3f3ced (0x9cee bytes) 000025[1] 1789583756.249122 read 0x206000 thru 0x214289 (0xe28a bytes) 000028[0] 1789583756.271160 mapwrite 0x53b000 thru 0x5433ed (0x83ee bytes) 000029[1] 1789583756.609733 read 0x223000 thru 0x2231db (0x1dc bytes) 000031[1] 1789583756.650095 trunc from 0x5433ee to 0x144819 (0x3febd4 bytes) 000032[0] 1789583756.715817 mapwrite 0x839000 thru 0x8410dd (0x80de bytes) 000033[1] 1789583756.904139 mapread 0x6f8000 thru 0x70060c (0x860d bytes) truncating to largest ever: 0x6d7466 000035[0] 1789583756.985652 trunc from 0x8410de to 0x6d7465 (0x169c78 bytes) 000037[1] 1789583757.105808 mapwrite 0x527000 thru 0x5322d6 (0xb2d7 bytes) 000039[1] 1789583757.349627 punch from 0x3a9cf9 to 0x3a9d03 (0xb bytes) truncating to largest ever: 0x973a49 000041[1] 1789583757.396178 trunc from 0x6d7466 to 0x973a48 (0x29c5e3 bytes) 000043[0] 1789583757.508818 write 0x4bd000 thru 0x4be831 (0x1832 bytes) 000044[0] 1789583757.546234 read 0x963000 thru 0x972a0e (0xfa0f bytes) 000045[0] 1789583757.616575 mapwrite 0x75b000 thru 0x768815 (0xd816 bytes) 000047[0] 1789583757.819558 mapread 0x1ca000 thru 0x1d3ca5 (0x9ca6 bytes) 000048[0] 1789583757.869751 write 0x780000 thru 0x784d27 (0x4d28 bytes) 000049[1] 1789583757.893143 write 0x165000 thru 0x16d549 (0x854a bytes) 000050[1] 1789583757.976220 trunc from 0x973a49 to 0x82e6f3 (0x145355 bytes) 000051[1] 1789583758.061039 write 0x61e000 thru 0x61e038 (0x39 bytes) 000052[0] 1789583758.074568 trunc from 0x82e6f4 to 0x93b127 (0x10ca34 bytes) 000053[1] 1789583758.120606 write 0x35a000 thru 0x364ab3 (0xaab4 bytes) 000055[0] 1789583758.182338 trunc from 0x93b128 to 0x6292e5 (0x311e42 bytes) 000056[1] 1789583758.274705 read 0x39d000 thru 0x3a17cd (0x47ce bytes) 000057[0] 1789583758.306149 mapwrite 0x9d0000 thru 0x9d9681 (0x9682 bytes) 000059[0] 1789583758.476191 mapread 0x4db000 thru 0x4e028c (0x528d bytes) 000060[1] 1789583758.499134 mapwrite 0x81a000 thru 0x829fad (0xffae bytes) 000064[1] 1789583758.773344 mapread 0x18e000 thru 0x18fd6c (0x1d6d bytes) 000065[1] 1789583758.822406 write 0x661000 thru 0x663a95 (0x2a96 bytes) 000066[0] 1789583758.880853 trunc from 0x9d9682 to 0x32a315 (0x6af36c bytes) 000068[1] 1789583759.048642 write 0x607000 thru 0x61474b (0xd74c bytes) 000069[1] 1789583759.086485 write 0x4a8000 thru 0x4b2342 (0xa343 bytes) 000070[1] 1789583759.124349 write 0x804000 thru 0x80e5ef (0xa5f0 bytes) 000071[0] 1789583759.131316 mapwrite 0x94a000 thru 0x94ebfb (0x4bfc bytes) 000072[0] 1789583759.318812 trunc from 0x94ebfc to 0x6cb861 (0x28339a bytes) 000073[1] 1789583759.380125 trunc from 0x6cb862 to 0x5703be (0x15b4a3 bytes) 000074[1] 1789583759.432594 read 0x302000 thru 0x30baba (0x9abb bytes) 000075[1] 1789583759.449297 mapwrite 0x1d5000 thru 0x1e3314 (0xe315 bytes) 000076[0] 1789583759.684251 mapread 0x0c2000 thru 0x0ced07 (0xcd08 bytes) 000079[0] 1789583759.761666 write 0x4b1000 thru 0x4b8fe1 (0x7fe2 bytes) 000082[1] 1789583759.807120 punch from 0x20f092 to 0x216123 (0x7092 bytes) 000083[0] 1789583759.835707 read 0x24f000 thru 0x259fb9 (0xafba bytes) 000084[0] 1789583759.856098 mapread 0x20d000 thru 0x210664 (0x3665 bytes) 000085[0] 1789583759.873841 read 0x358000 thru 0x363cb0 (0xbcb1 bytes) 000086[1] 1789583759.881237 trunc from 0x5703bf to 0x33c1dc (0x2341e2 bytes) 000087[1] 1789583759.931532 read 0x1d4000 thru 0x1d493d (0x93e bytes) 000088[1] 1789583760.006680 trunc from 0x33c1dd to 0x0a96ff (0x292add bytes) 000090[0] 1789583760.035516 mapread 0x08b000 thru 0x096622 (0xb623 bytes) 000091[0] 1789583760.079281 mapread 0x061000 thru 0x06ad79 (0x9d7a bytes) 000092[0] 1789583760.093236 trunc from 0x0a9700 to 0x71d099 (0x67399a bytes) 000093[1] 1789583760.126545 punch from 0x26be65 to 0x279595 (0xd731 bytes) 000094[0] 1789583760.145888 read 0x550000 thru 0x553246 (0x3247 bytes) 000096[1] 1789583760.179985 write 0x21a000 thru 0x228441 (0xe442 bytes) 000097[0] 1789583760.208229 write 0x455000 thru 0x461b04 (0xcb05 bytes) 000101[0] 1789583760.265300 trunc from 0x71d09a to 0x65a794 (0xc2905 bytes) 000102[0] 1789583760.284987 mapread 0x106000 thru 0x107951 (0x1952 bytes) 000103[1] 1789583760.320586 mapwrite 0x6b5000 thru 0x6bf781 (0xa782 bytes) 000104[0] 1789583760.524097 write 0x388000 thru 0x38b128 (0x3129 bytes) 000107[0] 1789583760.560237 write 0x1ca000 thru 0x1d12fc (0x72fd bytes) 000108[1] 1789583760.574157 write 0x54f000 thru 0x551cef (0x2cf0 bytes) 000109[1] 1789583760.609628 write 0x950000 thru 0x956b81 (0x6b82 bytes) 000110[1] 1789583760.612514 trunc from 0x956b82 to 0x25f1df (0x6f79a2 bytes) 000111[1] 1789583760.630646 read 0x02c000 thru 0x03876a (0xc76b bytes) 000112[0] 1789583760.705984 trunc from 0x25f1e0 to 0x297fb6 (0x38dd7 bytes) 000113[1] 1789583760.765156 trunc from 0x297fb7 to 0x3c11bb (0x129205 bytes) 000117[0] 1789583760.888265 read 0x291000 thru 0x291bc0 (0xbc1 bytes) 000118[0] 1789583760.917918 write 0x0f3000 thru 0x0fbb13 (0x8b14 bytes) 000122[0] 1789583760.972817 mapwrite 0x243000 thru 0x245f40 (0x2f41 bytes) 000124[0] 1789583761.200894 mapwrite 0x5c4000 thru 0x5c8348 (0x4349 bytes) 000128[0] 1789583761.427715 mapwrite 0x7dc000 thru 0x7e3e74 (0x7e75 bytes) 000129[1] 1789583761.698304 read 0x07a000 thru 0x0863cd (0xc3ce bytes) 000130[1] 1789583761.785543 mapwrite 0x612000 thru 0x614183 (0x2184 bytes) 000133[1] 1789583761.876503 read 0x795000 thru 0x79ffa3 (0xafa4 bytes) 000136[1] 1789583761.897404 mapread 0x5a0000 thru 0x5a1187 (0x1188 bytes) 000140[0] 1789583761.921995 read 0x72b000 thru 0x737f6c (0xcf6d bytes) 000141[0] 1789583761.953709 mapread 0x40b000 thru 0x40c8f7 (0x18f8 bytes) 000142[0] 1789583761.966239 mapread 0x7a6000 thru 0x7ad888 (0x7889 bytes) 000148[0] 1789583761.982820 read 0x109000 thru 0x10af4e (0x1f4f bytes) 000149[0] 1789583761.987643 mapwrite 0x0d5000 thru 0x0d8a29 (0x3a2a bytes) 000151[1] 1789583762.085298 write 0x99a000 thru 0x99cdaf (0x2db0 bytes) 000152[1] 1789583762.104923 read 0x3c4000 thru 0x3cc4f7 (0x84f8 bytes) 000154[0] 1789583762.142166 trunc from 0x99cdb0 to 0x5e4f2c (0x3b7e83 bytes) 000156[0] 1789583762.249624 trunc from 0x5e4f2d to 0x18624e (0x45ecde bytes) 000157[1] 1789583762.293086 mapread 0x039000 thru 0x045ba7 (0xcba8 bytes) 000158[1] 1789583762.336420 read 0x100000 thru 0x10896a (0x896b bytes) 000159[0] 1789583762.377067 write 0x93b000 thru 0x947632 (0xc633 bytes) 000160[1] 1789583762.422955 mapwrite 0x6ba000 thru 0x6bb131 (0x1132 bytes) 000161[1] 1789583762.678207 mapwrite 0x2fb000 thru 0x303cc9 (0x8cca bytes) 000162[0] 1789583762.758965 mapwrite 0x206000 thru 0x20e5fc (0x85fd bytes) 000163[1] 1789583762.916131 mapwrite 0x407000 thru 0x40e7b4 (0x77b5 bytes) 000164[1] 1789583763.050086 mapread 0x79b000 thru 0x7a6793 (0xb794 bytes) 000165[1] 1789583763.157177 mapread 0x077000 thru 0x0846d2 (0xd6d3 bytes) 000167[1] 1789583763.246438 punch from 0x90f39f to 0x910f33 (0x1b95 bytes) 000169[0] 1789583763.264882 mapread 0x4bb000 thru 0x4ca183 (0xf184 bytes) 000170[0] 1789583763.317465 trunc from 0x947633 to 0x0ea304 (0x85d32e bytes) 000171[1] 1789583763.369547 write 0x49e000 thru 0x4a7ba9 (0x9baa bytes) 000172[0] 1789583763.403757 trunc from 0x4a7baa to 0x933c52 (0x48c0a9 bytes) 000173[1] 1789583763.445926 mapwrite 0x025000 thru 0x0273b9 (0x23ba bytes) 000177[0] 1789583763.721654 write 0x339000 thru 0x34770a (0xe70b bytes) 000178[1] 1789583763.793485 mapwrite 0x140000 thru 0x143f2f (0x3f30 bytes) 000182[0] 1789583763.998476 mapread 0x85e000 thru 0x865827 (0x7828 bytes) 000183[0] 1789583764.047238 punch from 0x6ba55e to 0x6c4766 (0xa209 bytes) 000185[1] 1789583764.068615 mapread 0x533000 thru 0x53b4fc (0x84fd bytes) 000188[1] 1789583764.157104 mapread 0x29d000 thru 0x2a8b30 (0xbb31 bytes) 000189[0] 1789583764.298909 read 0x80b000 thru 0x80d95b (0x295c bytes) 000190[1] 1789583764.306560 punch from 0x486db3 to 0x496b85 (0xfdd3 bytes) 000191[1] 1789583764.377015 trunc from 0x933c53 to 0x652e9f (0x2e0db3 bytes) 000192[0] 1789583764.397345 mapwrite 0x0d8000 thru 0x0e49bf (0xc9c0 bytes) 000194[0] 1789583764.505022 mapwrite 0x4ad000 thru 0x4af5d5 (0x25d6 bytes) 000195[0] 1789583764.568673 trunc from 0x652ea0 to 0x5b09e8 (0xa24b7 bytes) 000196[1] 1789583764.587386 mapwrite 0x0ef000 thru 0x0fd49c (0xe49d bytes) truncating to largest ever: 0x989401 000198[0] 1789583764.774073 trunc from 0x5b09e9 to 0x989400 (0x3d8a18 bytes) 000199[0] 1789583764.801049 write 0x332000 thru 0x341912 (0xf913 bytes) 000200[0] 1789583764.833841 write 0x0c5000 thru 0x0c66a7 (0x16a8 bytes) 000201[1] 1789583764.867133 mapread 0x814000 thru 0x81bc6b (0x7c6c bytes) 000203[1] 1789583764.952530 mapread 0x5a6000 thru 0x5af8c4 (0x98c5 bytes) 000204[0] 1789583765.005587 write 0x785000 thru 0x78555d (0x55e bytes) 000205[0] 1789583765.047917 write 0x55c000 thru 0x55d4bd (0x14be bytes) 000206[1] 1789583765.060094 trunc from 0x989401 to 0x225fa3 (0x76345d bytes) 000208[0] 1789583765.131019 punch from 0x0c8fd1 to 0x0d22dd (0x930d bytes) 000209[0] 1789583765.164908 trunc from 0x225fa4 to 0x84d6bd (0x62771a bytes) 000210[0] 1789583765.199908 punch from 0x57f980 to 0x5824f2 (0x2b73 bytes) 000212[0] 1789583765.223012 mapwrite 0x456000 thru 0x457f02 (0x1f03 bytes) 000213[0] 1789583765.292599 read 0x479000 thru 0x479b5c (0xb5d bytes) 000214[0] 1789583765.303751 read 0x179000 thru 0x17e4ad (0x54ae bytes) 000217[1] 1789583765.347179 read 0x22a000 thru 0x232708 (0x8709 bytes) 000218[0] 1789583765.377208 read 0x28b000 thru 0x28f00a (0x400b bytes) 000219[0] 1789583765.411100 mapread 0x77d000 thru 0x789f1c (0xcf1d bytes) 000222[0] 1789583765.447569 read 0x781000 thru 0x788adf (0x7ae0 bytes) 000223[1] 1789583765.448089 write 0x742000 thru 0x750874 (0xe875 bytes) 000224[1] 1789583765.467436 punch from 0x822433 to 0x829500 (0x70ce bytes) 000225[0] 1789583765.484378 write 0x6a7000 thru 0x6ad701 (0x6702 bytes) 000226[0] 1789583765.552074 mapread 0x587000 thru 0x5968ef (0xf8f0 bytes) 000227[0] 1789583765.577492 punch from 0x27165f to 0x28047b (0xee1d bytes) 000231[0] 1789583765.600390 write 0x30f000 thru 0x31e4c3 (0xf4c4 bytes) 000232[0] 1789583765.609501 read 0x218000 thru 0x2200fd (0x80fe bytes) 000233[0] 1789583765.644322 write 0x357000 thru 0x3663b7 (0xf3b8 bytes) 000234[0] 1789583765.652655 trunc from 0x84d6be to 0x0314ae (0x81c20f bytes) 000235[0] 1789583765.678656 write 0x517000 thru 0x526547 (0xf548 bytes) 000237[1] 1789583765.707896 read 0x3d6000 thru 0x3e537d (0xf37e bytes) 000240[0] 1789583765.740091 mapwrite 0x7af000 thru 0x7b7b66 (0x8b67 bytes) 000242[0] 1789583765.953103 mapread 0x56c000 thru 0x571002 (0x5003 bytes) 000244[1] 1789583766.008965 trunc from 0x7b7b67 to 0x3dba88 (0x3dc0de bytes) 000245[0] 1789583766.057417 read 0x2e3000 thru 0x2f102c (0xe02d bytes) 000249[1] 1789583766.084594 mapread 0x292000 thru 0x2a071c (0xe71d bytes) 000253[1] 1789583766.107641 mapwrite 0x0d4000 thru 0x0d9145 (0x5146 bytes) 000254[1] 1789583766.176080 mapread 0x045000 thru 0x0525f7 (0xd5f8 bytes) 000255[1] 1789583766.204638 write 0x957000 thru 0x95a389 (0x338a bytes) 000258[0] 1789583766.209196 write 0x7a5000 thru 0x7b4dc2 (0xfdc3 bytes) 000260[1] 1789583766.260858 write 0x0a2000 thru 0x0b043e (0xe43f bytes) 000263[0] 1789583766.310839 mapwrite 0x6e6000 thru 0x6e6e2b (0xe2c bytes) 000265[1] 1789583766.405153 mapread 0x002000 thru 0x00f0e6 (0xd0e7 bytes) 000266[1] 1789583766.429991 write 0x44c000 thru 0x44eef8 (0x2ef9 bytes) 000267[1] 1789583766.481833 read 0x6e9000 thru 0x6f2b9a (0x9b9b bytes) 000268[1] 1789583766.490265 trunc from 0x95a38a to 0x7bac93 (0x19f6f6 bytes) 000272[0] 1789583766.527059 mapread 0x00e000 thru 0x0175fc (0x95fd bytes) 000273[1] 1789583766.572852 mapread 0x623000 thru 0x62dfa4 (0xafa5 bytes) 000274[1] 1789583766.600576 mapread 0x504000 thru 0x50f822 (0xb823 bytes) 000275[0] 1789583766.623165 read 0x1fc000 thru 0x1fe7f8 (0x27f9 bytes) 000276[1] 1789583766.629445 read 0x4f4000 thru 0x4fcc43 (0x8c44 bytes) 000277[0] 1789583766.642008 punch from 0x2e739c to 0x2ed402 (0x6067 bytes) 000279[0] 1789583766.669938 mapwrite 0x2ff000 thru 0x308592 (0x9593 bytes) 000280[0] 1789583766.717374 read 0x635000 thru 0x635de6 (0xde7 bytes) 000281[0] 1789583766.734191 trunc from 0x7bac94 to 0x04bf2f (0x76ed64 bytes) 000282[0] 1789583766.751217 read 0x023000 thru 0x023f6a (0xf6b bytes) 000283[1] 1789583766.778363 write 0x2d2000 thru 0x2dbc08 (0x9c09 bytes) 000284[1] 1789583766.804165 write 0x147000 thru 0x1538b2 (0xc8b3 bytes) 000285[0] 1789583766.810962 mapwrite 0x1c9000 thru 0x1ce9ec (0x59ed bytes) 000286[1] 1789583766.964903 write 0x503000 thru 0x50e7a8 (0xb7a9 bytes) 000287[1] 1789583766.985477 mapwrite 0x5d2000 thru 0x5d7f99 (0x5f9a bytes) 000289[1] 1789583767.057871 mapread 0x24c000 thru 0x25a356 (0xe357 bytes) 000290[1] 1789583767.104426 mapread 0x142000 thru 0x14cd8a (0xad8b bytes) 000291[1] 1789583767.137157 mapread 0x56f000 thru 0x57c419 (0xd41a bytes) 000297[1] 1789583767.225462 punch from 0x5c802d to 0x5cd2d9 (0x52ad bytes) 000298[0] 1789583767.246647 punch from 0x4003ee to 0x4016a8 (0x12bb bytes) 000302[0] 1789583767.275871 mapread 0x4f9000 thru 0x500e79 (0x7e7a bytes) 000303[0] 1789583767.378165 write 0x184000 thru 0x18ccef (0x8cf0 bytes) 000309[0] 1789583767.463463 mapwrite 0x614000 thru 0x623ec6 (0xfec7 bytes) 000314[1] 1789583767.610019 punch from 0x41647b to 0x418109 (0x1c8f bytes) 000315[1] 1789583767.648884 mapread 0x4ed000 thru 0x4fc45c (0xf45d bytes) 000316[0] 1789583767.679753 mapread 0x4a0000 thru 0x4ac33a (0xc33b bytes) 000321[0] 1789583767.770421 write 0x45d000 thru 0x468277 (0xb278 bytes) 000322[0] 1789583767.806756 punch from 0x271656 to 0x276345 (0x4cf0 bytes) 000323[0] 1789583767.825954 punch from 0x316c06 to 0x31f2f4 (0x86ef bytes) 000324[1] 1789583767.842929 mapwrite 0x93b000 thru 0x94319d (0x819e bytes) 000326[1] 1789583768.010444 mapwrite 0x4cc000 thru 0x4d93af (0xd3b0 bytes) 000329[0] 1789583768.139336 mapwrite 0x02b000 thru 0x0361d4 (0xb1d5 bytes) 000330[0] 1789583768.278892 mapwrite 0x9e9000 thru 0x9f3633 (0xa634 bytes) 000332[1] 1789583768.426851 mapwrite 0x21f000 thru 0x22e464 (0xf465 bytes) 000334[0] 1789583768.626676 trunc from 0x9f3634 to 0x1ee22f (0x805404 bytes) 000336[1] 1789583768.703802 mapwrite 0x8ce000 thru 0x8d65eb (0x85ec bytes) 000338[1] 1789583768.865625 punch from 0x348aad to 0x355411 (0xc965 bytes) 000339[0] 1789583768.900471 trunc from 0x8d65ec to 0x157ca3 (0x77e948 bytes) 000340[0] 1789583768.949861 read 0x035000 thru 0x03e848 (0x9849 bytes) 000341[0] 1789583768.991569 trunc from 0x157ca4 to 0x88ee8a (0x7371e7 bytes) 000343[0] 1789583769.115046 punch from 0x0f8b41 to 0x103eb4 (0xb374 bytes) 000344[1] 1789583769.146456 trunc from 0x88ee8b to 0x910f4f (0x820c5 bytes) 000351[0] 1789583769.319523 write 0x515000 thru 0x5225d5 (0xd5d6 bytes) 000353[0] 1789583769.374425 mapwrite 0x1b3000 thru 0x1b9c8f (0x6c90 bytes) 000354[1] 1789583769.431987 write 0x119000 thru 0x11adfa (0x1dfb bytes) 000355[0] 1789583769.459104 write 0x5e9000 thru 0x5eba76 (0x2a77 bytes) 000356[1] 1789583769.493990 mapread 0x8c6000 thru 0x8c7950 (0x1951 bytes) 000358[1] 1789583769.535674 mapwrite 0x38c000 thru 0x39af79 (0xef7a bytes) 000359[0] 1789583769.638989 mapwrite 0x1a0000 thru 0x1a6351 (0x6352 bytes) 000360[1] 1789583769.783493 read 0x494000 thru 0x49fd82 (0xbd83 bytes) 000361[1] 1789583769.806461 mapread 0x59a000 thru 0x5a425c (0xa25d bytes) 000362[0] 1789583769.823030 mapwrite 0x155000 thru 0x16432b (0xf32c bytes) 000363[1] 1789583769.937741 mapread 0x79f000 thru 0x7a8cf7 (0x9cf8 bytes) 000367[0] 1789583769.980663 mapread 0x073000 thru 0x078e5c (0x5e5d bytes) 000368[0] 1789583770.003449 read 0x8ef000 thru 0x8f1b9b (0x2b9c bytes) 000369[1] 1789583770.008930 write 0x5ad000 thru 0x5b9b61 (0xcb62 bytes) 000370[1] 1789583770.035334 read 0x700000 thru 0x706021 (0x6022 bytes) 000371[0] 1789583770.042784 punch from 0x509769 to 0x50c890 (0x3128 bytes) 000373[1] 1789583770.087818 punch from 0x6d7d93 to 0x6de5f0 (0x685e bytes) 000374[1] 1789583770.127718 mapread 0x66e000 thru 0x66e23b (0x23c bytes) 000375[1] 1789583770.151989 mapwrite 0x7a3000 thru 0x7aff6f (0xcf70 bytes) 000378[0] 1789583770.281694 mapread 0x0bb000 thru 0x0c997a (0xe97b bytes) 000379[0] 1789583770.341968 mapwrite 0x81a000 thru 0x81c95d (0x295e bytes) 000380[1] 1789583770.431012 trunc from 0x910f50 to 0x4775ed (0x499962 bytes) 000381[1] 1789583770.491067 mapread 0x3a4000 thru 0x3a598a (0x198b bytes) 000384[0] 1789583770.531068 trunc from 0x4775ee to 0x4ccfc9 (0x559dc bytes) 000385[0] 1789583770.572727 punch from 0x1d3c63 to 0x1de590 (0xa92e bytes) 000390[1] 1789583770.648921 mapwrite 0x6f6000 thru 0x6fa750 (0x4751 bytes) 000391[0] 1789583770.829721 write 0x012000 thru 0x016220 (0x4221 bytes) 000392[0] 1789583770.874814 mapwrite 0x689000 thru 0x68a4b3 (0x14b4 bytes) 000394[0] 1789583770.945344 read 0x6a5000 thru 0x6b22b5 (0xd2b6 bytes) 000395[1] 1789583770.954470 write 0x2bd000 thru 0x2be64f (0x1650 bytes) 000399[0] 1789583771.001979 mapread 0x3d0000 thru 0x3da837 (0xa838 bytes) 000400[1] 1789583771.044067 read 0x668000 thru 0x66a57b (0x257c bytes) 000401[1] 1789583771.052328 trunc from 0x6fa751 to 0x80157f (0x106e2f bytes) 000403[1] 1789583771.107423 mapread 0x0ca000 thru 0x0d9e44 (0xfe45 bytes) 000404[0] 1789583771.140492 mapread 0x3d8000 thru 0x3e60f6 (0xe0f7 bytes) 000406[1] 1789583771.186646 write 0x2da000 thru 0x2dd5d2 (0x35d3 bytes) 000408[0] 1789583771.228941 trunc from 0x801580 to 0x134fdb (0x6cc5a4 bytes) 000409[0] 1789583771.278891 punch from 0x0e9fe6 to 0x0ecfe3 (0x2ffe bytes) 000410[1] 1789583771.299617 read 0x09c000 thru 0x0a9e14 (0xde15 bytes) 000415[1] 1789583771.350895 mapread 0x08e000 thru 0x0912d0 (0x32d1 bytes) 000418[0] 1789583771.352089 mapread 0x07d000 thru 0x08a8b1 (0xd8b2 bytes) 000423[1] 1789583771.373961 mapread 0x03a000 thru 0x03b60b (0x160c bytes) 000428[0] 1789583771.375462 mapwrite 0x5cb000 thru 0x5ce248 (0x3249 bytes) 000429[1] 1789583771.480403 mapread 0x34f000 thru 0x35e44c (0xf44d bytes) 000431[0] 1789583771.592697 mapread 0x329000 thru 0x32a96a (0x196b bytes) 000434[0] 1789583771.607725 mapwrite 0x494000 thru 0x4a2887 (0xe888 bytes) 000435[0] 1789583771.705147 write 0x3b3000 thru 0x3beece (0xbecf bytes) 000437[0] 1789583771.738010 write 0x25e000 thru 0x25ef0e (0xf0f bytes) 000438[0] 1789583771.763546 write 0x1b5000 thru 0x1bc2c0 (0x72c1 bytes) 000439[1] 1789583771.785527 read 0x279000 thru 0x281fe1 (0x8fe2 bytes) 000440[0] 1789583771.828920 trunc from 0x5ce249 to 0x824e7c (0x256c34 bytes) 000441[0] 1789583771.863247 mapread 0x001000 thru 0x0045fd (0x35fe bytes) 000442[1] 1789583771.881126 punch from 0x46bccd to 0x46d2f6 (0x162a bytes) 000443[0] 1789583771.901630 mapwrite 0x59f000 thru 0x59f45f (0x460 bytes) 000446[0] 1789583772.004259 mapwrite 0x4e5000 thru 0x4f3876 (0xe877 bytes) 000447[1] 1789583772.080743 mapread 0x529000 thru 0x52ec83 (0x5c84 bytes) 000448[1] 1789583772.114671 read 0x7d3000 thru 0x7d6e90 (0x3e91 bytes) 000450[0] 1789583772.124065 write 0x363000 thru 0x364288 (0x1289 bytes) 000452[0] 1789583772.166984 read 0x53c000 thru 0x5416bf (0x56c0 bytes) 000453[0] 1789583772.174039 punch from 0x07faae to 0x08051c (0xa6f bytes) 000454[0] 1789583772.192128 mapwrite 0x4c5000 thru 0x4d15c4 (0xc5c5 bytes) 000455[0] 1789583772.265048 punch from 0x6c07fa to 0x6c0e95 (0x69c bytes) 000456[1] 1789583772.280949 read 0x6d2000 thru 0x6de095 (0xc096 bytes) 000458[1] 1789583772.305878 mapwrite 0x418000 thru 0x4224b1 (0xa4b2 bytes) 000463[1] 1789583772.409055 punch from 0x1d7f06 to 0x1d9db9 (0x1eb4 bytes) 000465[1] 1789583772.435157 write 0x04f000 thru 0x056329 (0x732a bytes) 000466[1] 1789583772.452444 read 0x200000 thru 0x20bb7a (0xbb7b bytes) 000467[0] 1789583772.504003 write 0x3dc000 thru 0x3dd471 (0x1472 bytes) 000468[1] 1789583772.542628 write 0x1e4000 thru 0x1ec321 (0x8322 bytes) 000469[0] 1789583772.580847 write 0x89e000 thru 0x8aa6fe (0xc6ff bytes) 000471[0] 1789583772.600111 mapwrite 0x4be000 thru 0x4c58fe (0x78ff bytes) 000473[1] 1789583772.672377 punch from 0x04fbcc to 0x05640e (0x6843 bytes) 000474[1] 1789583772.690042 mapwrite 0x51d000 thru 0x52222c (0x522d bytes) 000475[1] 1789583772.752836 mapwrite 0x0f1000 thru 0x100ef4 (0xfef5 bytes) 000477[0] 1789583772.857882 punch from 0x34c41b to 0x34ea8a (0x2670 bytes) 000478[0] 1789583772.875921 punch from 0x5018d5 to 0x50ac1f (0x934b bytes) 000479[0] 1789583772.894874 mapwrite 0x6ab000 thru 0x6afadb (0x4adc bytes) 000480[1] 1789583773.012201 mapwrite 0x658000 thru 0x65b028 (0x3029 bytes) 000481[0] 1789583773.113933 read 0x288000 thru 0x294f4c (0xcf4d bytes) 000483[1] 1789583773.149018 punch from 0x34e8a1 to 0x35678d (0x7eed bytes) 000485[0] 1789583773.234417 write 0x3ac000 thru 0x3b7775 (0xb776 bytes) 000488[1] 1789583773.289451 trunc from 0x8aa6ff to 0x5e5321 (0x2c53dd bytes) 000490[0] 1789583773.409736 write 0x122000 thru 0x1275de (0x55df bytes) truncating to largest ever: 0x9a8d9a 000491[1] 1789583773.447836 trunc from 0x5e5322 to 0x9a8d99 (0x3c3a78 bytes) 000492[0] 1789583773.534088 mapwrite 0x5a3000 thru 0x5ac040 (0x9041 bytes) 000493[1] 1789583773.873173 write 0x998000 thru 0x9a3922 (0xb923 bytes) 000495[0] 1789583773.899755 write 0x416000 thru 0x418b09 (0x2b0a bytes) 000497[0] 1789583773.962917 read 0x5b2000 thru 0x5b843c (0x643d bytes) 000499[0] 1789583773.980924 punch from 0x033e5f to 0x03f044 (0xb1e6 bytes) 000500[0] 1789583773.989411 read 0x8ce000 thru 0x8dcb67 (0xeb68 bytes) All operations completed A-OK! PASS 16a (60s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 14:36:20 (1789583780) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.404328 s, 20.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.291853 s, 28.7 MB/s Chance of close/open is 1 in 50 Seed set to 3784 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1789583785.535385 write 0x33f367 thru 0x33fae0 (0x77a bytes) 000005[1] 1789583785.636520 write 0x562ff8 thru 0x5630b4 (0xbd bytes) 000006[1] 1789583785.665749 read 0x4ff969 thru 0x500569 (0xc01 bytes) 000007[1] 1789583785.687114 read 0x17bbb5 thru 0x17c045 (0x491 bytes) 000008[1] 1789583785.721183 write 0x6bde3f thru 0x6be4dc (0x69e bytes) 000009[1] 1789583785.722660 punch from 0x385157 to 0x385325 (0x1cf bytes) 000014[0] 1789583785.796749 read 0x335d65 thru 0x33753b (0x17d7 bytes) 000015[1] 1789583785.822150 write 0x9ed27f thru 0x9ee394 (0x1116 bytes) 000017[1] 1789583785.849602 mapwrite 0x9f2241 thru 0x9f2c95 (0xa55 bytes) 000018[0] 1789583785.908065 punch from 0x5ef477 to 0x5ef5b0 (0x13a bytes) 000020[0] 1789583785.953546 mapread 0x0d19ed thru 0x0d25a6 (0xbba bytes) truncating to largest ever: 0x68c19b 000021[0] 1789583785.980737 trunc from 0x9f2c96 to 0x68c19a (0x366afb bytes) 000022[1] 1789583786.030375 mapread 0x60cba1 thru 0x60dc1a (0x107a bytes) 000023[0] 1789583786.073146 read 0x5b09f0 thru 0x5b0cdd (0x2ee bytes) 000024[1] 1789583786.083856 mapread 0x249fee thru 0x24aa83 (0xa96 bytes) 000026[1] 1789583786.113021 write 0x133998 thru 0x134534 (0xb9d bytes) 000027[0] 1789583786.157454 punch from 0x468040 to 0x469f6a (0x1f2b bytes) 000030[1] 1789583786.228532 mapread 0x00146a thru 0x0033a3 (0x1f3a bytes) 000031[0] 1789583786.273745 write 0x7f68a2 thru 0x7f7108 (0x867 bytes) 000032[1] 1789583786.316249 read 0x68137c thru 0x6830f6 (0x1d7b bytes) 000033[0] 1789583786.438925 punch from 0x087d24 to 0x089393 (0x1670 bytes) 000034[1] 1789583786.489290 write 0x18f373 thru 0x18fb8d (0x81b bytes) 000037[1] 1789583786.537120 write 0x11389f thru 0x1149c6 (0x1128 bytes) 000038[1] 1789583786.547370 mapread 0x717037 thru 0x7173ec (0x3b6 bytes) 000039[0] 1789583786.580517 mapwrite 0x5847c9 thru 0x58584e (0x1086 bytes) 000041[1] 1789583786.685736 mapwrite 0x7ece19 thru 0x7eeac6 (0x1cae bytes) 000042[0] 1789583786.777408 punch from 0x32da28 to 0x32eac5 (0x109e bytes) 000043[0] 1789583786.809098 punch from 0x791010 to 0x791962 (0x953 bytes) 000045[0] 1789583786.859994 write 0x515e03 thru 0x516519 (0x717 bytes) 000046[1] 1789583786.886123 trunc from 0x7f7109 to 0x3658b0 (0x491858 bytes) 000048[0] 1789583786.935575 mapread 0x3091f9 thru 0x30b187 (0x1f8f bytes) 000050[0] 1789583786.996215 read 0x18ac9e thru 0x18aff5 (0x358 bytes) 000051[0] 1789583787.046719 write 0x958c09 thru 0x959dbc (0x11b4 bytes) 000053[1] 1789583787.124994 punch from 0x52d784 to 0x52d863 (0xe0 bytes) 000055[1] 1789583787.160215 mapread 0x1fec46 thru 0x2008be (0x1c79 bytes) 000056[1] 1789583787.208898 trunc from 0x959dbd to 0x2e1666 (0x678756 bytes) 000057[0] 1789583787.243388 mapread 0x1b6fcf thru 0x1b8773 (0x17a5 bytes) 000063[1] 1789583787.334843 mapwrite 0x6926ac thru 0x693d70 (0x16c5 bytes) 000064[1] 1789583787.432202 write 0x004dab thru 0x0063a7 (0x15fd bytes) 000065[0] 1789583787.472590 write 0x56055d thru 0x56074c (0x1f0 bytes) 000066[0] 1789583787.504139 mapread 0x223ce2 thru 0x224b87 (0xea6 bytes) 000067[1] 1789583787.564467 read 0x153c28 thru 0x155a50 (0x1e29 bytes) 000069[0] 1789583787.582982 trunc from 0x693d71 to 0x4212dc (0x272a94 bytes) truncating to largest ever: 0x8c3a1f 000071[0] 1789583787.641540 trunc from 0x4212dd to 0x8c3a1e (0x4a2742 bytes) 000072[1] 1789583787.684787 mapwrite 0x927c32 thru 0x927c88 (0x57 bytes) 000073[0] 1789583787.802352 read 0x7283a9 thru 0x728ba2 (0x7fa bytes) 000074[1] 1789583787.844614 mapread 0x47327c thru 0x474e33 (0x1bb8 bytes) 000075[0] 1789583787.885525 punch from 0x66dca7 to 0x66f4b2 (0x180c bytes) 000078[0] 1789583787.949175 read 0x35dc49 thru 0x35f725 (0x1add bytes) 000079[1] 1789583787.964753 mapwrite 0x6df227 thru 0x6df379 (0x153 bytes) 000080[1] 1789583788.028712 mapwrite 0x00daad thru 0x00f581 (0x1ad5 bytes) 000081[1] 1789583788.119396 read 0x5d08a0 thru 0x5d132d (0xa8e bytes) 000082[1] 1789583788.126975 trunc from 0x927c89 to 0x2b24ec (0x67579c bytes) 000084[1] 1789583788.187819 trunc from 0x2b24ed to 0x13cb41 (0x1759ab bytes) 000085[1] 1789583788.205671 mapwrite 0x284bd8 thru 0x2861de (0x1607 bytes) 000086[1] 1789583788.297402 mapread 0x23a527 thru 0x23b6f1 (0x11cb bytes) 000088[1] 1789583788.340989 punch from 0x0b7ea6 to 0x0b9e1f (0x1f7a bytes) 000089[0] 1789583788.349074 mapread 0x1c0db4 thru 0x1c1aa2 (0xcef bytes) 000090[1] 1789583788.389619 read 0x215fdb thru 0x2167a3 (0x7c9 bytes) truncating to largest ever: 0x9a3e5b 000091[0] 1789583788.456709 trunc from 0x2861df to 0x9a3e5a (0x71dc7c bytes) 000093[1] 1789583788.538286 read 0x17ec06 thru 0x17f3cd (0x7c8 bytes) 000094[1] 1789583788.561682 write 0x695ddd thru 0x6960c7 (0x2eb bytes) 000095[1] 1789583788.594143 mapwrite 0x90970e thru 0x90b142 (0x1a35 bytes) 000097[0] 1789583788.727862 mapread 0x07c824 thru 0x07e1e0 (0x19bd bytes) 000098[0] 1789583788.777681 read 0x13f6f8 thru 0x140eeb (0x17f4 bytes) 000099[0] 1789583788.793134 trunc from 0x9a3e5b to 0x1e6a57 (0x7bd403 bytes) 000100[1] 1789583788.840062 mapwrite 0x259bbc thru 0x25a020 (0x465 bytes) 000101[1] 1789583788.963118 write 0x547565 thru 0x54922c (0x1cc8 bytes) 000109[1] 1789583789.058399 trunc from 0x54922d to 0x34e715 (0x1fab17 bytes) 000112[0] 1789583789.151334 mapwrite 0x5748b0 thru 0x57548e (0xbdf bytes) 000113[0] 1789583789.243341 read 0x2aec3f thru 0x2afa7f (0xe41 bytes) 000115[0] 1789583789.301380 mapread 0x13d269 thru 0x13d4da (0x272 bytes) 000118[1] 1789583789.374300 read 0x165d76 thru 0x1664ed (0x778 bytes) 000120[0] 1789583789.393658 punch from 0x1feeca to 0x2000a5 (0x11dc bytes) 000123[1] 1789583789.411524 read 0x24fb22 thru 0x251843 (0x1d22 bytes) 000124[0] 1789583789.519675 read 0x1f878c thru 0x1fa076 (0x18eb bytes) 000125[1] 1789583789.537846 mapwrite 0x2ec039 thru 0x2ecb9a (0xb62 bytes) 000128[1] 1789583789.657546 trunc from 0x57548f to 0x731547 (0x1bc0b9 bytes) 000131[1] 1789583789.730566 write 0x3da38d thru 0x3da764 (0x3d8 bytes) 000132[0] 1789583789.735754 read 0x4b65da thru 0x4b8342 (0x1d69 bytes) 000133[0] 1789583789.757126 read 0x661186 thru 0x6627f9 (0x1674 bytes) truncating to largest ever: 0x9bef5f 000136[1] 1789583789.794326 trunc from 0x731548 to 0x9bef5e (0x28da17 bytes) 000138[1] 1789583789.867085 read 0x58c289 thru 0x58d829 (0x15a1 bytes) 000140[1] 1789583789.899363 mapwrite 0x8aaba8 thru 0x8ab5c2 (0xa1b bytes) 000142[0] 1789583789.977637 write 0x58b40b thru 0x58d28e (0x1e84 bytes) 000147[0] 1789583790.069751 write 0x583a6b thru 0x584302 (0x898 bytes) 000148[0] 1789583790.097205 mapwrite 0x4f627e thru 0x4f7931 (0x16b4 bytes) 000149[0] 1789583790.181521 read 0x5e4b9f thru 0x5e5e3c (0x129e bytes) 000150[1] 1789583790.200362 punch from 0x827aa7 to 0x829139 (0x1693 bytes) 000151[1] 1789583790.226146 write 0x676960 thru 0x676e85 (0x526 bytes) 000152[0] 1789583790.270446 mapwrite 0x24929b thru 0x249f51 (0xcb7 bytes) 000153[0] 1789583790.348053 mapread 0x245e85 thru 0x247820 (0x199c bytes) 000155[1] 1789583790.391675 punch from 0x5f102e to 0x5f2796 (0x1769 bytes) 000158[0] 1789583790.431225 trunc from 0x9bef5f to 0x0ddee5 (0x8e1079 bytes) 000159[0] 1789583790.466193 write 0x11a1fc thru 0x11b9da (0x17df bytes) 000161[1] 1789583790.493015 write 0x5525cc thru 0x552ff6 (0xa2b bytes) 000162[1] 1789583790.524706 trunc from 0x552ff7 to 0x6e208c (0x18f096 bytes) 000169[0] 1789583790.638377 mapread 0x2e7e44 thru 0x2e8af0 (0xcad bytes) 000174[0] 1789583790.702414 trunc from 0x6e208d to 0x5cd405 (0x114c87 bytes) 000177[0] 1789583790.818678 trunc from 0x5cd406 to 0x5872de (0x46127 bytes) 000180[1] 1789583790.883617 trunc from 0x5872df to 0x51f662 (0x67c7c bytes) 000181[0] 1789583790.934939 write 0x5482b3 thru 0x5491c9 (0xf17 bytes) 000182[1] 1789583790.979339 write 0x85bbe6 thru 0x85c35e (0x779 bytes) 000183[1] 1789583791.002079 trunc from 0x85c35f to 0x23248f (0x629ecf bytes) 000185[0] 1789583791.052427 mapwrite 0x8ca110 thru 0x8cb00d (0xefe bytes) 000187[0] 1789583791.182634 mapwrite 0x4d83ee thru 0x4d9df2 (0x1a05 bytes) 000189[1] 1789583791.254464 trunc from 0x8cb00e to 0x2e43f0 (0x5e6c1d bytes) 000190[1] 1789583791.301327 punch from 0x26fc8d to 0x270c5a (0xfce bytes) 000191[0] 1789583791.330408 trunc from 0x2e43f1 to 0x3ac320 (0xc7f30 bytes) 000193[1] 1789583791.407683 punch from 0x022bc2 to 0x023c45 (0x1084 bytes) 000194[1] 1789583791.440666 mapwrite 0x77439e thru 0x774ad3 (0x736 bytes) 000195[0] 1789583791.557581 read 0x012ae7 thru 0x013c3f (0x1159 bytes) 000196[0] 1789583791.587608 trunc from 0x774ad4 to 0x46e0e0 (0x3069f3 bytes) 000198[0] 1789583791.651864 read 0x12d67e thru 0x12f1bd (0x1b40 bytes) 000200[0] 1789583791.679062 read 0x46d40e thru 0x46e0e0 (0xcd3 bytes) 000201[1] 1789583791.703368 read 0x35a2d8 thru 0x35bcf6 (0x1a1f bytes) 000202[0] 1789583791.723507 mapwrite 0x0d4002 thru 0x0d5a1d (0x1a1c bytes) 000204[0] 1789583791.794132 read 0x3c2fde thru 0x3c4f13 (0x1f36 bytes) 000205[0] 1789583791.803109 mapwrite 0x8f6686 thru 0x8f73a9 (0xd24 bytes) 000206[0] 1789583791.891113 mapread 0x56782d thru 0x56864b (0xe1f bytes) 000207[1] 1789583791.902030 read 0x00200c thru 0x0033a9 (0x139e bytes) 000211[0] 1789583792.026605 mapread 0x4417ad thru 0x44198c (0x1e0 bytes) 000213[0] 1789583792.043736 read 0x4ffd43 thru 0x500687 (0x945 bytes) 000214[0] 1789583792.061187 trunc from 0x8f73aa to 0x4fefa8 (0x3f8401 bytes) 000215[0] 1789583792.129687 read 0x292ea4 thru 0x293e9c (0xff9 bytes) 000216[0] 1789583792.155052 read 0x26ab16 thru 0x26b21a (0x705 bytes) 000218[1] 1789583792.183697 mapwrite 0x214991 thru 0x214a9c (0x10c bytes) 000220[1] 1789583792.286124 read 0x1d9ddf thru 0x1d9ea2 (0xc4 bytes) 000223[1] 1789583792.340531 punch from 0x0e9c1f to 0x0eb6e0 (0x1ac2 bytes) 000225[1] 1789583792.376457 read 0x1b84c1 thru 0x1b9c5a (0x179a bytes) 000227[0] 1789583792.408584 mapwrite 0x6dfdbf thru 0x6e0c2f (0xe71 bytes) 000229[1] 1789583792.510482 trunc from 0x6e0c30 to 0x090679 (0x6505b6 bytes) 000230[1] 1789583792.552424 read 0x033e83 thru 0x035b80 (0x1cfe bytes) 000234[1] 1789583792.612145 read 0x065b47 thru 0x065bfb (0xb5 bytes) 000236[0] 1789583792.645174 read 0x08ca57 thru 0x08d7f0 (0xd9a bytes) 000237[0] 1789583792.671938 trunc from 0x09067a to 0x8b4c1e (0x8245a5 bytes) 000238[0] 1789583792.748651 mapread 0x4c2f2a thru 0x4c4016 (0x10ed bytes) 000241[0] 1789583792.907171 mapwrite 0x7c5377 thru 0x7c5e03 (0xa8d bytes) 000242[1] 1789583793.004936 write 0x07e80b thru 0x0804d0 (0x1cc6 bytes) 000243[0] 1789583793.113347 write 0x91f39f thru 0x92013a (0xd9c bytes) 000244[0] 1789583793.134598 write 0x5bc597 thru 0x5bdc6f (0x16d9 bytes) 000245[0] 1789583793.164708 read 0x3b8809 thru 0x3b9e3b (0x1633 bytes) 000246[0] 1789583793.192049 read 0x218712 thru 0x218769 (0x58 bytes) 000249[0] 1789583793.252508 mapwrite 0x2a6453 thru 0x2a68fc (0x4aa bytes) 000250[0] 1789583793.322726 trunc from 0x92013b to 0x797224 (0x188f16 bytes) 000251[1] 1789583793.359166 trunc from 0x797225 to 0x8aa0f6 (0x112ed2 bytes) 000256[0] 1789583793.469397 mapread 0x7b9638 thru 0x7bb424 (0x1ded bytes) 000260[1] 1789583793.553186 trunc from 0x8aa0f7 to 0x55f976 (0x34a780 bytes) 000262[1] 1789583793.618353 trunc from 0x55f977 to 0x44dced (0x111c89 bytes) 000265[1] 1789583793.715261 mapread 0x139e0e thru 0x13a78e (0x981 bytes) 000266[0] 1789583793.733967 mapread 0x3029bf thru 0x303b33 (0x1175 bytes) 000267[1] 1789583793.776291 trunc from 0x44dcee to 0x3a4cb8 (0xa9035 bytes) 000268[1] 1789583793.821288 punch from 0x38fe15 to 0x3917ae (0x199a bytes) 000270[1] 1789583793.882594 trunc from 0x3a4cb9 to 0x49ddf5 (0xf913d bytes) 000273[0] 1789583794.011934 mapread 0x237486 thru 0x237c0b (0x786 bytes) 000274[0] 1789583794.119038 punch from 0x1208ee to 0x121d2e (0x1441 bytes) 000276[0] 1789583794.227089 mapwrite 0x133e19 thru 0x1358fc (0x1ae4 bytes) 000278[0] 1789583794.449178 mapread 0x192192 thru 0x194140 (0x1faf bytes) 000279[0] 1789583794.475639 mapwrite 0x77e2bb thru 0x77f5e0 (0x1326 bytes) 000280[0] 1789583794.550744 mapwrite 0x17339a thru 0x174227 (0xe8e bytes) 000281[1] 1789583794.592717 mapwrite 0x0d441c thru 0x0d4cfe (0x8e3 bytes) 000282[1] 1789583794.670506 mapwrite 0x49f79a thru 0x49fc9e (0x505 bytes) 000286[1] 1789583794.842540 read 0x28c55c thru 0x28c8cf (0x374 bytes) 000288[1] 1789583794.876533 read 0x39880f thru 0x39a0d2 (0x18c4 bytes) 000289[1] 1789583794.904114 write 0x426887 thru 0x42886d (0x1fe7 bytes) 000290[1] 1789583794.906017 mapwrite 0x202544 thru 0x202999 (0x456 bytes) 000292[0] 1789583794.986722 write 0x001e1d thru 0x002e72 (0x1056 bytes) 000293[0] 1789583795.031970 mapread 0x076648 thru 0x077998 (0x1351 bytes) 000294[0] 1789583795.072440 mapread 0x4d9c0d thru 0x4da175 (0x569 bytes) 000295[1] 1789583795.125195 trunc from 0x77f5e1 to 0x5ad077 (0x1d2569 bytes) 000296[0] 1789583795.176956 mapread 0x17d20d thru 0x17f01c (0x1e10 bytes) 000298[0] 1789583795.223286 read 0x2beb1d thru 0x2bfff1 (0x14d5 bytes) 000300[1] 1789583795.314389 read 0x0ed17c thru 0x0ee360 (0x11e5 bytes) 000302[1] 1789583795.332681 punch from 0x222860 to 0x223b6c (0x130d bytes) 000303[0] 1789583795.355534 read 0x57e1df thru 0x57f0e1 (0xf03 bytes) 000304[0] 1789583795.374008 mapwrite 0x4c2002 thru 0x4c2bf8 (0xbf7 bytes) 000306[0] 1789583795.482019 trunc from 0x5ad078 to 0x26cb95 (0x3404e2 bytes) 000307[0] 1789583795.531464 mapwrite 0x2f9234 thru 0x2fb051 (0x1e1e bytes) 000308[1] 1789583795.657359 write 0x224e56 thru 0x22508a (0x235 bytes) 000310[0] 1789583795.718991 trunc from 0x2fb052 to 0x289d7a (0x712d7 bytes) 000312[1] 1789583795.845582 read 0x1cde1e thru 0x1ceee7 (0x10ca bytes) 000315[0] 1789583795.998089 punch from 0x00ca35 to 0x00cdc2 (0x38e bytes) 000317[1] 1789583796.073327 mapwrite 0x2f6371 thru 0x2f742a (0x10ba bytes) 000319[0] 1789583796.262691 trunc from 0x2f742b to 0x05721a (0x2a0210 bytes) 000320[1] 1789583796.333989 trunc from 0x05721b to 0x7ce130 (0x776f16 bytes) 000321[0] 1789583796.435045 read 0x2b2f4c thru 0x2b4460 (0x1515 bytes) 000322[0] 1789583796.547980 read 0x425fa4 thru 0x426000 (0x5d bytes) 000326[1] 1789583796.598633 write 0x61f42c thru 0x620d84 (0x1959 bytes) 000327[0] 1789583796.712379 punch from 0x0c6138 to 0x0c7f18 (0x1de1 bytes) 000328[0] 1789583796.755274 trunc from 0x7ce131 to 0x5b2ba9 (0x21b587 bytes) 000329[0] 1789583796.793528 trunc from 0x5b2baa to 0x4a1001 (0x111ba8 bytes) 000330[0] 1789583796.818378 write 0x2b8c00 thru 0x2babda (0x1fdb bytes) 000332[1] 1789583796.847970 mapread 0x101253 thru 0x10226d (0x101b bytes) 000333[0] 1789583796.895133 mapwrite 0x4b1756 thru 0x4b2ebb (0x1766 bytes) 000334[1] 1789583797.003723 write 0x4c90fc thru 0x4caba4 (0x1aa9 bytes) 000335[0] 1789583797.045915 write 0x64ac2e thru 0x64bb90 (0xf63 bytes) 000336[1] 1789583797.076700 mapwrite 0x9aa19e thru 0x9ac174 (0x1fd7 bytes) 000338[0] 1789583797.239035 write 0x0ef752 thru 0x0f0456 (0xd05 bytes) 000339[0] 1789583797.279997 mapread 0x876a4c thru 0x878460 (0x1a15 bytes) 000341[0] 1789583797.328100 read 0x564812 thru 0x565f5a (0x1749 bytes) 000342[1] 1789583797.345884 write 0x000c44 thru 0x001759 (0xb16 bytes) 000343[1] 1789583797.378444 punch from 0x0906d7 to 0x090e2a (0x754 bytes) 000344[1] 1789583797.387393 mapread 0x338312 thru 0x33a20f (0x1efe bytes) 000345[1] 1789583797.414727 read 0x6080ac thru 0x6085b2 (0x507 bytes) 000347[1] 1789583797.436551 mapwrite 0x9d3499 thru 0x9d4fee (0x1b56 bytes) 000350[0] 1789583797.578509 punch from 0x2c0632 to 0x2c25a4 (0x1f73 bytes) 000351[0] 1789583797.614868 punch from 0x9a6abb to 0x9a7d55 (0x129b bytes) 000354[1] 1789583797.662486 punch from 0x4f6100 to 0x4f62e4 (0x1e5 bytes) 000355[0] 1789583797.681051 punch from 0x81c952 to 0x81cf49 (0x5f8 bytes) 000357[0] 1789583797.718124 mapwrite 0x1263d1 thru 0x127744 (0x1374 bytes) 000358[0] 1789583797.792144 trunc from 0x9d4fef to 0x2d21f8 (0x702df6 bytes) 000359[0] 1789583797.849897 trunc from 0x2d21f9 to 0x3d222d (0x100035 bytes) 000360[0] 1789583797.871030 punch from 0x201813 to 0x2020ac (0x89a bytes) 000363[0] 1789583797.907558 mapread 0x1ab72e thru 0x1ab83d (0x110 bytes) 000365[0] 1789583797.921190 mapread 0x008e2f thru 0x00a432 (0x1604 bytes) 000366[1] 1789583797.952268 read 0x1d3b69 thru 0x1d3f19 (0x3b1 bytes) 000368[0] 1789583798.094167 mapwrite 0x818453 thru 0x81979a (0x1348 bytes) 000371[1] 1789583798.192137 write 0x6f1365 thru 0x6f1f8b (0xc27 bytes) 000373[0] 1789583798.238112 mapwrite 0x5126a7 thru 0x512824 (0x17e bytes) 000374[0] 1789583798.297909 mapread 0x2bc902 thru 0x2be05f (0x175e bytes) 000377[0] 1789583798.331960 punch from 0x163d53 to 0x1643f0 (0x69e bytes) 000382[0] 1789583798.405588 mapwrite 0x8ec597 thru 0x8ee197 (0x1c01 bytes) 000383[0] 1789583798.479012 mapread 0x2b2d4f thru 0x2b3ddb (0x108d bytes) 000385[1] 1789583798.488469 mapwrite 0x519dbc thru 0x51a9ee (0xc33 bytes) 000386[0] 1789583798.564541 mapwrite 0x10c959 thru 0x10e835 (0x1edd bytes) 000390[1] 1789583798.710239 write 0x75e403 thru 0x760350 (0x1f4e bytes) 000391[1] 1789583798.759652 read 0x053320 thru 0x05472a (0x140b bytes) 000393[1] 1789583798.809012 write 0x8924ff thru 0x89375a (0x125c bytes) 000394[1] 1789583798.841274 punch from 0x3e3df8 to 0x3e4568 (0x771 bytes) 000395[1] 1789583798.878839 read 0x363e51 thru 0x3655ab (0x175b bytes) 000396[0] 1789583798.885891 write 0x6f9aa3 thru 0x6f9d44 (0x2a2 bytes) 000398[1] 1789583798.947827 mapwrite 0x720164 thru 0x720f07 (0xda4 bytes) 000399[1] 1789583799.021414 write 0x64261e thru 0x643993 (0x1376 bytes) 000401[0] 1789583799.058472 mapwrite 0x5c08ec thru 0x5c25c0 (0x1cd5 bytes) 000402[0] 1789583799.180353 punch from 0x0ad77f to 0x0ade17 (0x699 bytes) 000406[1] 1789583799.244756 mapwrite 0x6b094b thru 0x6b0ecd (0x583 bytes) 000409[0] 1789583799.346928 mapwrite 0x5ec438 thru 0x5eccd6 (0x89f bytes) 000410[0] 1789583799.463017 read 0x28fc16 thru 0x290c6f (0x105a bytes) 000412[0] 1789583799.502260 read 0x0b3fb8 thru 0x0b4f22 (0xf6b bytes) 000418[0] 1789583799.598996 write 0x3c6959 thru 0x3c86b2 (0x1d5a bytes) 000419[1] 1789583799.612390 write 0x0a8e16 thru 0x0a9bfe (0xde9 bytes) 000420[0] 1789583799.647442 read 0x671827 thru 0x6728b5 (0x108f bytes) 000424[0] 1789583799.704400 mapread 0x53d44b thru 0x53e424 (0xfda bytes) 000425[1] 1789583799.736011 read 0x1939f4 thru 0x195567 (0x1b74 bytes) 000426[0] 1789583799.759104 mapread 0x32793f thru 0x329135 (0x17f7 bytes) 000427[0] 1789583799.808679 write 0x9907c6 thru 0x990f26 (0x761 bytes) 000429[0] 1789583799.841449 trunc from 0x990f27 to 0x54690c (0x44a61a bytes) 000430[1] 1789583799.861313 punch from 0x1dfa54 to 0x1e15c4 (0x1b71 bytes) 000431[1] 1789583799.890134 mapread 0x3822cf thru 0x38344f (0x1181 bytes) 000432[0] 1789583799.916562 trunc from 0x54690d to 0x956c64 (0x410358 bytes) 000433[1] 1789583799.973170 write 0x3678de thru 0x36969d (0x1dc0 bytes) 000435[1] 1789583800.021877 write 0x398d14 thru 0x399e6b (0x1158 bytes) 000436[1] 1789583800.043110 trunc from 0x956c65 to 0x80c566 (0x14a6fe bytes) 000439[1] 1789583800.098080 punch from 0x53b7ec to 0x53c8b6 (0x10cb bytes) 000440[1] 1789583800.123452 punch from 0x4e7045 to 0x4e8e24 (0x1de0 bytes) 000441[0] 1789583800.156030 trunc from 0x80c567 to 0x99f900 (0x19339a bytes) 000442[1] 1789583800.223925 mapwrite 0x9883b3 thru 0x988cc7 (0x915 bytes) 000444[0] 1789583800.316371 punch from 0x6d5e49 to 0x6d7cce (0x1e86 bytes) 000445[1] 1789583800.339908 mapwrite 0x71cfba thru 0x71e6ad (0x16f4 bytes) 000448[0] 1789583800.458183 read 0x21eb4c thru 0x220557 (0x1a0c bytes) 000450[0] 1789583800.495035 mapwrite 0x286fa6 thru 0x288a6e (0x1ac9 bytes) 000451[0] 1789583800.576483 write 0x489f69 thru 0x48bb16 (0x1bae bytes) 000452[1] 1789583800.618663 punch from 0x366598 to 0x367957 (0x13c0 bytes) 000453[1] 1789583800.648019 mapread 0x4d5b7a thru 0x4d7191 (0x1618 bytes) 000458[1] 1789583800.747425 trunc from 0x99f901 to 0x55029a (0x44f666 bytes) truncating to largest ever: 0x9d6ed0 000459[1] 1789583800.811619 trunc from 0x55029b to 0x9d6ecf (0x486c35 bytes) 000460[1] 1789583800.835744 mapwrite 0x05567f thru 0x0566f7 (0x1079 bytes) 000461[1] 1789583800.879400 read 0x34cd27 thru 0x34d984 (0xc5e bytes) 000462[0] 1789583800.884941 mapwrite 0x95ccf5 thru 0x95e875 (0x1b81 bytes) 000463[0] 1789583800.985919 mapread 0x7fd81c thru 0x7fe998 (0x117d bytes) 000464[1] 1789583801.014002 mapwrite 0x101dc2 thru 0x1034a8 (0x16e7 bytes) 000465[1] 1789583801.095935 mapread 0x57aec8 thru 0x57b21e (0x357 bytes) 000466[1] 1789583801.116245 trunc from 0x9d6ed0 to 0x64561c (0x3918b3 bytes) 000469[1] 1789583801.149904 mapread 0x3fecb3 thru 0x400b68 (0x1eb6 bytes) 000479[1] 1789583801.279383 mapwrite 0x22652c thru 0x227fb5 (0x1a8a bytes) 000482[0] 1789583801.411263 mapread 0x2dc8c1 thru 0x2de261 (0x19a1 bytes) 000483[0] 1789583801.475649 write 0x06170f thru 0x0636fb (0x1fed bytes) 000484[0] 1789583801.530304 read 0x2584b4 thru 0x258bd4 (0x721 bytes) 000485[1] 1789583801.551715 mapwrite 0x18c650 thru 0x18d7fb (0x11ac bytes) 000486[0] 1789583801.681280 write 0x17e412 thru 0x17f016 (0xc05 bytes) 000489[1] 1789583801.760718 write 0x1c5d71 thru 0x1c6170 (0x400 bytes) 000490[1] 1789583801.784833 mapwrite 0x769dbb thru 0x76b6ef (0x1935 bytes) 000491[1] 1789583801.862163 write 0x24043e thru 0x241348 (0xf0b bytes) 000492[0] 1789583801.899625 read 0x451b00 thru 0x453766 (0x1c67 bytes) 000493[0] 1789583801.925472 mapread 0x303058 thru 0x3036d5 (0x67e bytes) 000496[1] 1789583801.978737 mapwrite 0x8bab37 thru 0x8bb5be (0xa88 bytes) 000499[1] 1789583802.178434 mapwrite 0x9c6ae7 thru 0x9c76ca (0xbe4 bytes) All operations completed A-OK! PASS 16b (28s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 14:36:48 (1789583808) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 14:36:52 (1789583812) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.34588 s, 24.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.269467 s, 31.1 MB/s ldlm.namespaces.MGC192.168.201.136@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8aa484c95000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8aa492786800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8aa484c95000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff8aa492786800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8aa492786800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8aa492786800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.89865 s, 13.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.96613 s, 35.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.07976 s, 34.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.51826 s, 13.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.0314 s, 7.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.27076 s, 19.9 MB/s PASS 16d (55s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 14:37:47 (1789583867) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.490912 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172893 s, 6.1 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 14:37:56 (1789583876) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 14:38:25 (1789583905) TIMEOUT PASS 16g (30s) == sanityn test 16h: mmap read after truncate file ======= 14:38:55 (1789583935) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (9s) == sanityn test 16i: read after truncate file ============ 14:39:04 (1789583944) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (10s) == sanityn test 16j: race dio with buffered i/o ========== 14:39:14 (1789583954) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.67115 s, 15.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.8741 s, 14.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.4093 s, 5.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.7585 s, 9.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.60102 s, 7.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.37661 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.74154 s, 9.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.57295 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.02161 s, 8.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.405 s, 6.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.46963 s, 5.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.26995 s, 11.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.70044 s, 9.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.19666 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.35662 s, 7.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.16069 s, 8.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.35424 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.47589 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.7707 s, 7.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (45s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 14:39:59 (1789583999) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Wed Sep 16 14:40:46 EDT 2026 + fsx 2 done: rc=0 Wed Sep 16 14:40:46 EDT 2026 + fsx 3 done: rc=0 Wed Sep 16 14:40:46 EDT 2026 + fsx 4 done: rc=0 Wed Sep 16 14:40:46 EDT 2026 + fsx 5 done: rc=0 Wed Sep 16 14:40:46 EDT 2026 PASS 16k (57s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 14:40:56 (1789584056) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 14:41:09 (1789584069) mmap test1: basic mmap operation (PASS, 0.21298s) mmap test2: MAP_PRIVATE not write back (PASS, 0.095063s) mmap test3: concurrent mmap ops on two nodes (PASS, 6.3769s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.7889s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.26056s) mmap test6: check mmap write/read content on two nodes (PASS, 0.18809s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.4897s) mmap test11: multi-thread shared mmap (PASS, 1.0765s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1197s) PASS 18 (24s) == sanityn test 19: test concurrent uncached read races ========================================================================= 14:41:34 (1789584094) oleg136-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (6s) == sanityn test 20: test extra readahead page left in cache ============================================================== 14:41:39 (1789584099) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d20.sanityn PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 14:41:47 (1789584107) striped dir -i1 -c2 -H all_char /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (9s) == sanityn test 23: others should see updated atime while another read============================================================== 14:41:56 (1789584116) now is 1789584119 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 new atime is 1789584180 PASS 23 (69s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 14:43:05 (1789584185) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 516.2K 405 515.8K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 518.7K 335 518.4K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 114.5K 467 114.1K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 115.4K 466 114.9K 1% /mnt/lustre[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 516.2K 405 515.8K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 518.7K 335 518.4K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 114.5K 467 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.4K 466 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.4M 2.1G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.6G 33.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 528612 405 528207 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 531165 335 530830 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 117267 467 116800 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118162 466 117696 1% /mnt/lustre2[OST:1] filesystem_summary: 235236 740 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 516.2K 405 515.8K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 518.7K 335 518.4K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 114.5K 467 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.4K 466 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.7K 740 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 528612 405 528207 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 531165 335 530830 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 117267 467 116800 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118162 466 117696 1% /mnt/lustre[OST:1] filesystem_summary: 235236 740 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 528612 405 528207 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 531165 335 530830 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 117267 467 116800 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118162 466 117696 1% /mnt/lustre2[OST:1] filesystem_summary: 235236 740 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 14:43:12 (1789584192) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 14:43:19 (1789584199) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (6s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 14:43:25 (1789584205) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (7s) == sanityn test 26a: allow mtime to get older ============ 14:43:32 (1789584212) utime: good mknod times 1789584212 <= 1789584213 <= 1789584213 for /mnt/lustre/f26a utime: good mknod times 1789584212 <= 1789584213 <= 1789584213 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 14:43:39 (1789584219) PASS 26b (11s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 14:43:50 (1789584230) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279599 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-16 14:43:52.000000000 -0400 Birth: 2026-09-16 14:43:51.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279599 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-16 14:43:52.000000000 -0400 Birth: 2026-09-16 14:43:51.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 14:44:00 (1789584240) striped dir -i0 -c2 -H crush /mnt/lustre/d30.sanityn 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 14:44:10 (1789584250) striped dir -i1 -c2 -H crush2 /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 14:44:18 (1789584258) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg136-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete striped dir -i1 -c2 -H all_char /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.136@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 14:44:39 (1789584279) -rw-r--r-- 1 root root 0 Sep 16 14:44 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 14:44:49 (1789584289) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 45198 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 14:44:56 (1789584296) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 14:45:04 (1789584304) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 14:45:08 (1789584308) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.45962 s, 35.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.32285 s, 39.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.67176 s, 31.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.80793 s, 29.0 MB/s contention_seconds 4, contention_events 14 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.01155 s, 26.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.30178 s, 22.8 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.39705 s, 37.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50007 s, 35.0 MB/s PASS 32c (30s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 14:45:40 (1789584340) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (5s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 14:45:45 (1789584345) Stopping /mnt/lustre-mds1 (opts:) on oleg136-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (53s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 14:46:38 (1789584398) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 1 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (58s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 14:47:36 (1789584456) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 1.53 seconds: 65.29 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 1.39 seconds: 71.90 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1789584463 ; total 0 ; last 0) total: 100 unlinks in 5 seconds: 20.000000 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1789584468 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (20s) == sanityn test 34: no lock timeout under IO ============= 14:47:56 (1789584476) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8aa484c95000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8aa484c95000.ost_server_uuid in IDLE state after 0 sec oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8aa484c95000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8aa484c95000.ost_server_uuid in IDLE state after 0 sec oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8aa484c95000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8aa484c95000.ost_server_uuid in IDLE state after 0 sec oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8aa484c95000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8aa484c95000.ost_server_uuid in IDLE state after 0 sec oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8aa484c95000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8aa484c95000.ost_server_uuid in IDLE state after 0 sec oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8aa484c95000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8aa484c95000.ost_server_uuid in IDLE state after 0 sec PASS 34 (135s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 14:50:12 (1789584612) striped dir -i1 -c2 -H crush2 /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 57223 57238 for 60 sec... - open/close 951 (time 1789584630.23 total 10.00 last 95.08) - open/close 1972 (time 1789584640.24 total 20.01 last 102.00) - open/close 2882 (time 1789584650.25 total 30.02 last 90.94) - open/close 3676 (time 1789584660.29 total 40.06 last 79.09) total: 4000 open/close in 43.72 seconds: 91.50 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.136@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.201.136@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (74s) == sanityn test 36: handle ESTALE/open-unlink correctly == 14:51:26 (1789584686) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg136-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.96846 s, 26.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.59002 s, 33.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.73298 s, 30.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7496704) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.50276 s, 34.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.80664 s, 29.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.09804 s, 25.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.16352 s, 45.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(6) *** before(7489536) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48613 s, 35.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(7) *** before(7492608) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.68018 s, 31.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7496704) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.63298 s, 32.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7496704) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.78983 s, 29.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7445504) after(7496704) PASS 36 (296s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 14:56:22 (1789584982) striped dir -i1 -c2 -H all_char /mnt/lustre/d37.sanityn multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 - create 1401 (time 1789584998.17 total 10.01 last 139.96) - create 2762 (time 1789585008.19 total 20.03 last 135.86) - create 4267 (time 1789585018.19 total 30.03 last 150.49) - create 5808 (time 1789585028.19 total 40.03 last 154.03) - create 7291 (time 1789585038.19 total 50.03 last 148.30) - create 9052 (time 1789585048.20 total 60.04 last 175.91) total: 10000 create in 65.07 seconds: 153.69 ops/second PASS 37 (78s) == sanityn test 39a: file mtime does not change after rename ========================================================== 14:57:40 (1789585060) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 14:57:48 (1789585068) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 14:57:56 (1789585076) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 14:58:05 (1789585085) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 14:58:15 (1789585095) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (20s) == sanityn test 40c: pdirops: link and others ======================================================================== 14:58:35 (1789585115) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 14:58:53 (1789585133) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (16s) == sanityn test 40e: pdirops: rename and others ======================================================================== 14:59:09 (1789585149) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 14:59:26 (1789585166) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (11s) == sanityn test 41b: pdirops: create vs create ======================================================================== 14:59:37 (1789585177) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 14:59:50 (1789585190) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 15:00:04 (1789585204) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 15:00:19 (1789585219) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 15:00:32 (1789585232) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 15:00:44 (1789585244) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 15:00:57 (1789585257) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 15:01:11 (1789585271) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1094s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 15:19:24 (1789586364) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 15:19:36 (1789586376) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 42b (17s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 15:19:53 (1789586393) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (17s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 15:20:10 (1789586410) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42d (15s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 15:20:25 (1789586425) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 15:20:39 (1789586439) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42f (13s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 15:20:53 (1789586453) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=3 Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 15:21:07 (1789586467) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 15:21:21 (1789586481) PASS 43a (174s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 15:24:15 (1789586655) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43b (15s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 15:24:30 (1789586670) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 15:24:44 (1789586684) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 15:24:59 (1789586699) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 15:25:12 (1789586712) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 15:25:26 (1789586726) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 15:25:40 (1789586740) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43h (16s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 15:25:56 (1789586756) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 43i (14s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 15:26:10 (1789586770) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (130s) == sanityn test 43k: unlink vs create ==================== 15:28:20 (1789586900) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1295s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 15:49:55 (1789588195) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Sep 16 15:50:04 EDT 2026 PASS 44a (15s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 15:50:10 (1789588210) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 15:50:25 (1789588225) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 15:50:38 (1789588238) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 15:50:53 (1789588253) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44e (13s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 15:51:06 (1789588266) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44f (16s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 15:51:23 (1789588283) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44g (16s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 15:51:39 (1789588299) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 15:51:54 (1789588314) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f44i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f44i.sanityn': File exists Conflict PASS 44i (12s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 15:52:06 (1789588326) PASS 45a (220s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 15:55:47 (1789588547) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45b (12s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 15:55:59 (1789588559) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45c (15s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 15:56:13 (1789588573) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (14s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 15:56:27 (1789588587) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45e (16s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 15:56:43 (1789588603) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (15s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 15:56:58 (1789588618) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 15:57:13 (1789588633) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 15:57:25 (1789588645) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 45i (13s) == sanityn test 45j: read vs rename ====================== 15:57:39 (1789588659) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1311s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 16:19:30 (1789589970) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (16s) == sanityn test 46b: pdirops: link vs create ======================================================================== 16:19:46 (1789589986) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 46b (19s) == sanityn test 46c: pdirops: link vs link ======================================================================== 16:20:05 (1789590005) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (14s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 16:20:19 (1789590019) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46d (16s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 16:20:35 (1789590035) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 16:20:49 (1789590049) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46f (14s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 16:21:03 (1789590063) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 16:21:18 (1789590078) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 16:21:32 (1789590092) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 lfs mkdir: dirstripe error on '/mnt/lustre2/f46i.sanityn': stripe already set lfs setdirstripe: cannot create dir '/mnt/lustre2/f46i.sanityn': File exists Conflict PASS 46i (13s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 16:21:46 (1789590106) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mkdir: cannot create directory '/mnt/lustre2/f47a.sanityn': File exists Conflict PASS 47a (14s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 16:22:00 (1789590120) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 open: File exists Conflict PASS 47b (12s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 16:22:12 (1789590132) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 link: cannot create link '/mnt/lustre2/f47c.sanityn' to '/mnt/lustre2/f47c.sanityn-2': File exists Conflict PASS 47c (14s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 16:22:26 (1789590146) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47d (11s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 16:22:37 (1789590157) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 mv: cannot overwrite directory '/mnt/lustre2/f47e.sanityn' with non-directory Conflict PASS 47e (10s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 16:22:47 (1789590167) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47f (12s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 16:22:59 (1789590179) ldlm.namespaces.mdt-lustre-MDT0000_UUID.lock_count=1 Conflict PASS 47g (16s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 16:23:15 (1789590195) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.264285 s, 38.7 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 16:23:27 (1789590207) 0+1 records in 0+1 records out 158 bytes copied, 0.0218205 s, 7.2 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (236542) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 16:23:37 (1789590217) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0165023 s, 62.1 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758938 s, 13.8 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 16:23:57 (1789590237) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0713294 s, 14.4 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 16:24:10 (1789590250) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156335 s, 6.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (10s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 16:24:20 (1789590260) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1325 0x52d 0x280000401 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 1325 0x52d 0x280000401 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 16:24:31 (1789590271) llite.lustre-ffff8aa484c95000.xattr_cache=1 llite.lustre-ffff8aa492786800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762667 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 16:24:33.000000000 -0400 Modify: 2026-09-16 16:24:33.000000000 -0400 Change: 2026-09-16 16:24:33.000000000 -0400 Birth: 2026-09-16 16:24:32.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281277 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 16:24:39.000000000 -0400 Modify: 2026-09-16 16:24:40.000000000 -0400 Change: 2026-09-16 16:24:40.000000000 -0400 Birth: 2026-09-16 16:24:39.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762670 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 16:24:46.000000000 -0400 Modify: 2026-09-16 16:24:46.000000000 -0400 Change: 2026-09-16 16:24:46.000000000 -0400 Birth: 2026-09-16 16:24:45.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281278 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 16:24:46.000000000 -0400 Modify: 2026-09-16 16:24:47.000000000 -0400 Change: 2026-09-16 16:24:47.000000000 -0400 Birth: 2026-09-16 16:24:45.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129603798762671 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 16:24:52.000000000 -0400 Modify: 2026-09-16 16:24:54.000000000 -0400 Change: 2026-09-16 16:24:54.000000000 -0400 Birth: 2026-09-16 16:24:52.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205289281280 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 16:24:52.000000000 -0400 Modify: 2026-09-16 16:24:54.000000000 -0400 Change: 2026-09-16 16:24:54.000000000 -0400 Birth: 2026-09-16 16:24:52.000000000 -0400 PASS 54 (35s) == sanityn test 55a: rename vs unlink target dir ========= 16:25:05 (1789590305) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (12s) == sanityn test 55b: rename vs unlink source dir ========= 16:25:17 (1789590317) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 16:25:31 (1789590331) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (18s) == sanityn test 55d: rename file vs link ================= 16:25:49 (1789590349) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (13s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 16:26:02 (1789590362) fail_loc=0x155 We survived after AB/BA race test fail_loc=0x0 PASS 55e (19s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 16:26:21 (1789590381) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 16:26:40 (1789590400) mdt.lustre-MDT0000.enable_rename_trylock=1 mdt.lustre-MDT0001.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 16:27:02 (1789590422) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789590423 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.622848 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (105s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 16:28:47 (1789590527) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789590532 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.968843 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (141s) == sanityn test 60: Verify data_version behaviour ======== 16:31:08 (1789590668) striped dir -i0 -c2 -H all_char /mnt/lustre/d60.sanityn PASS 60 (11s) == sanityn test 70a: cd directory && rm directory ======== 16:31:20 (1789590680) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 16:31:29 (1789590689) PASS 70b (14s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 16:31:43 (1789590703) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00973481 s, 6.7 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 16:31:47 (1789590707) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0177176 s, 3.7 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 16:31:51 (1789590711) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 16:31:55 (1789590715) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0516821 s, 1.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 16:31:59 (1789590719) llite.lustre-ffff8aa484c95000.xattr_cache=1 llite.lustre-ffff8aa492786800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 16:32:07 (1789590727) llite.lustre-ffff8aa484c95000.xattr_cache=1 llite.lustre-ffff8aa492786800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 16:32:14 (1789590734) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0150558 s, 68.0 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0256241 s, 40.0 kB/s 249863: get lock2 249865: get lock1 249865: done 249865: get lock2 249863: done 249863: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 249863: cannot get lock1: Resource deadlock avoided 249865: done 249865: exit rc=0 249863: done 249863: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 16:32:26 (1789590746) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0663337 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043456 s, 24.1 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 250455 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 16:32:39 (1789590759) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d76.sanityn opening files: ................ closing files: ................ PASS 76 (182s) == sanityn test 77a: check FIFO NRS policy =============== 16:35:41 (1789590941) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.43463 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068343 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040064 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787343 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125345 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223608 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191732 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185787 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224005 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242645 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205605 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280694 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254697 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278533 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309285 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287267 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341626 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727611 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105785 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756853 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833023 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804303 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867506 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915232 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938115 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940504 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673841 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105841 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111603 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109181 s, 9.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111792 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117392 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149462 s, 7.0 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 16:35:50 (1789590950) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.855163 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111904 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117423 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169769 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190704 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241168 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239489 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256369 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324058 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319324 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302571 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845633 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609989 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769084 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07737 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786181 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838445 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318907 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399617 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105204 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100731 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384031 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101514 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995747 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140891 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837026 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120806 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108913 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415522 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434158 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165303 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453299 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116605 s, 89.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.529674 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788128 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122391 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16337 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222482 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254781 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25245 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194562 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227206 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322784 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258661 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133229 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189998 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144193 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279826 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321539 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282967 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244196 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527899 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887079 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390754 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638063 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444994 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488788 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12608 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535234 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130437 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107813 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678171 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708584 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086107 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873203 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899217 s, 11.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (14s) == sanityn test 77c: check ORR NRS policy ================ 16:36:05 (1789590965) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.60363 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966997 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124373 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148068 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134662 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221601 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868902 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156546 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358743 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112267 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223609 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373739 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349701 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267736 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125282 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168828 s, 6.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254584 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059467 s, 17.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758611 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808084 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437412 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908753 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10296 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935559 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887227 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417373 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129348 s, 8.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0069525 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461474 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102083 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572377 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582108 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670321 s, 15.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.569489 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775112 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808362 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107306 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151416 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115399 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133189 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174795 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788132 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784872 s, 13.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355651 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317396 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280405 s, 3.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266253 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255888 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358631 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403561 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452842 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351433 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10849 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287236 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230067 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239645 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109107 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128813 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375176 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134856 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113705 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13128 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975968 s, 10.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146333 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156521 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151049 s, 6.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.614774 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787667 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103261 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102514 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230733 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254004 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154927 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305287 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211345 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400127 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241758 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384306 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227907 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197429 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495979 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431127 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.59956 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714287 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595064 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656925 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133356 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698831 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973226 s, 10.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896679 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132165 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958749 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137252 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927335 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919898 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901901 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771164 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148442 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186905 s, 56.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (18s) == sanityn test 77d: check TRR nrs policy ================ 16:36:23 (1789590983) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.580195 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132354 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130808 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219666 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211528 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215028 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262265 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185731 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302492 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210182 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302187 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275842 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420494 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425338 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351804 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42825 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472551 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869612 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598126 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653049 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105914 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062938 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278561 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119972 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125149 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895641 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956432 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036185 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109761 s, 9.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157183 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105079 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196168 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315127 s, 33.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.559353 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647211 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131806 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182032 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167629 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22311 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156778 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206691 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175349 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117419 s, 8.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192995 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418041 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36606 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466292 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353757 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363496 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347227 s, 3.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073892 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138161 s, 75.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179243 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083324 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244942 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951769 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707537 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565278 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124323 s, 84.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478174 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373752 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00962403 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611405 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720875 s, 14.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824259 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191137 s, 54.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.508774 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708491 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107284 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136023 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167909 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139177 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146526 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106594 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196967 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174304 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234751 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205152 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2334 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204375 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222512 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250149 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180391 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531126 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238375 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251512 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414357 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390004 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402687 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00814328 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148628 s, 70.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453281 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105513 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279933 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269152 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162636 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191699 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398355 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296217 s, 35.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 16:36:41 (1789591001) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.36@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.49523 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363605 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714937 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120483 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155833 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173358 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162247 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200557 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184435 s, 5.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228701 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227517 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261574 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204053 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216896 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231246 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242017 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317401 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16863 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168722 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112661 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404064 s, 26.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600705 s, 17.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072144 s, 14.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196236 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173831 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188639 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690974 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204886 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620884 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698153 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111935 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204636 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111349 s, 9.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.504208 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112271 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117681 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105292 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293222 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300568 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292432 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3809 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37181 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335869 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399357 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348132 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345558 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400036 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44977 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443438 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00228476 s, 459 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194971 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14485 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136432 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00244866 s, 428 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429184 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165957 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884703 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579955 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225117 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228506 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10121 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821528 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1471 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128495 s, 81.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231384 s, 45.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.617459 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561681 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129805 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103627 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187973 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240075 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238905 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126344 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106318 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102084 s, 10.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367616 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334351 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303953 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435552 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383847 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385104 s, 2.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41438 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382015 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00886045 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209204 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143031 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422919 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253895 s, 413 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00537951 s, 195 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00585185 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00896595 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00721354 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00248677 s, 422 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502679 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156206 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771924 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122506 s, 85.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243821 s, 43.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.650099 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870838 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953713 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14608 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205747 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113395 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21559 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265124 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116906 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13309 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416159 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438457 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350927 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366953 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360298 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429243 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.48949 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146822 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210331 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151409 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919684 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167486 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997304 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110754 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12327 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132056 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147059 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140245 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133468 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146694 s, 7.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147578 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150712 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17154 s, 6.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (26s) == sanityn test 77f: check TBF JobID nrs policy ========== 16:37:07 (1789591027) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.546316 s, 30.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923454 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104291 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111906 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189061 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121261 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17044 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161591 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649736 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309135 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190748 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178226 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302653 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318708 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379159 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404863 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267429 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110968 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821354 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110443 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452754 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534146 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500697 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13634 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112607 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092808 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788619 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119982 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945218 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12239 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108738 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126587 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105076 s, 10.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.680734 s, 24.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676074 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837475 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762227 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103685 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133594 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219985 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249626 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227801 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262812 s, 4.0 MB/s 1+0 records in 1+0 records out execvp fails running declare (13): Permission denied 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251531 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238116 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256168 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275961 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27455 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328694 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354008 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240613 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431989 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339129 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214725 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708594 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971136 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707238 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748932 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122211 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076498 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814215 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129599 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104084 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873629 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783389 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112345 s, 9.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.385769 s, 43.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255173 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060252 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597788 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608924 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781043 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114255 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133567 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133822 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189353 s, 5.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352003 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450883 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132912 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.497643 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186214 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148959 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529683 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.504867 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170143 s, 61.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925272 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227557 s, 46.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358523 s, 29.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.537211 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113783 s, 92.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818135 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243833 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.592807 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106871 s, 98.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469079 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300613 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247466 s, 424 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00273161 s, 384 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199214 s, 52.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.897699 s, 18.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113748 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158865 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110758 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179769 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196672 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242961 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437549 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44823 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.551417 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500888 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.655534 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.615641 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.693795 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.559236 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.749888 s, 1.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167416 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22795 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150243 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174456 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200636 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138002 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137692 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155562 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254926 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107326 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200614 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146251 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207706 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147241 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325322 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.885034 s, 1.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330756 s, 31.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (31s) == sanityn test 77g: Change TBF type directly ============ 16:37:39 (1789591059) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (13s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 16:37:51 (1789591071) oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg136-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg136-client: oleg136-server: ssh exited with exit code 2 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg136-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg136-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg136-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 PASS 77h (13s) == sanityn test 77i: Change rank of TBF rule ============= 16:38:05 (1789591085) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (23s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 16:38:27 (1789591107) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.613901 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086732 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18147 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120774 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235809 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202996 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241073 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173063 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17564 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24638 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26934 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320185 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108276 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229511 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265569 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251809 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358431 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333401 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00909397 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409583 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00606417 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903655 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985075 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0062049 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161419 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150578 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245954 s, 426 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139664 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751491 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179462 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161518 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120211 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104901 s, 100 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.78406 s, 13.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6409 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (62s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 16:39:29 (1789591169) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg136-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg136-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg136-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg136-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30849 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587204 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125441 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136486 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147177 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113825 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139293 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17889 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878813 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15757 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179385 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23843 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291646 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243603 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305762 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279445 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339977 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925958 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112844 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120482 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06783 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103305 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136456 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943194 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106298 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133887 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128873 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146492 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104157 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106188 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109473 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122058 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125133 s, 8.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0202 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0025 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg136-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg136-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg136-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg136-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27324 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157753 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167712 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159385 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169871 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21433 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223197 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227641 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192895 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221443 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24506 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208511 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236573 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235854 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252827 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335632 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373433 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109455 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864021 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10863 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11609 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929272 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117463 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114823 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114667 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117628 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115907 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123788 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119626 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125936 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138347 s, 7.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135802 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162657 s, 6.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9679 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9806 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (156s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 16:42:05 (1789591325) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg136-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg136-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg136-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg136-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31507 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133239 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606514 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152688 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211617 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24574 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288957 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290617 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275457 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311371 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287135 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388196 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143894 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346805 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370952 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376276 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480298 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600714 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557132 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062086 s, 16.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537249 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542557 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630998 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471654 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588405 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461225 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358912 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549699 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968754 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063454 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696149 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672762 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472539 s, 22.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1276 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9599 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg136-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg136-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg136-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg136-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30303 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13973 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144087 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119565 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148024 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186997 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194479 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19316 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206239 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197428 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224856 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218807 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291877 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288371 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301251 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294506 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353899 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05161 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933652 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105541 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119025 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983436 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134821 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927406 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101348 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110782 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10904 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117707 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141021 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139808 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124932 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139961 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15476 s, 6.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9096 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9387 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (152s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 16:44:37 (1789591477) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.469785 s, 35.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814489 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107356 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572355 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222776 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242502 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226032 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2797 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212314 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179426 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279426 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254557 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188327 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243526 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260379 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255083 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304537 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00287286 s, 365 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299453 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00244736 s, 428 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666102 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259505 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287013 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258453 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264324 s, 39.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285983 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030992 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317235 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329598 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324432 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370194 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405124 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378305 s, 27.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.464 s, 14.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9641 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.36@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.36@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.53966 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134372 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819474 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112785 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729963 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979019 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112267 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186716 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118865 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295202 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237002 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320821 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320142 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351439 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353976 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24581 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293023 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151986 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201417 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147979 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165422 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214131 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2015 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221883 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171658 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178887 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251625 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22877 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258981 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25106 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229936 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253366 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266582 s, 3.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=10 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.08815 s, 14.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96304 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.36@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.472894 s, 35.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079896 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984085 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106771 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974363 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287547 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262412 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318148 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289899 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318262 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315266 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349647 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359631 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37346 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312411 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324052 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362749 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040114 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519558 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078854 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452853 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573135 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070468 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05704 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703755 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169871 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889942 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840048 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991783 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865139 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991075 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116901 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130307 s, 8.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.28167 s, 12.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.67038 s, 18.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.580297 s, 28.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860319 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104754 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195437 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248408 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2452 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23313 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229058 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277924 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222061 s, 4.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279398 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198466 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293695 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212307 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33221 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286446 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259089 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465374 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318137 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578448 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614366 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685879 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414848 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446845 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399331 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044036 s, 23.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474873 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951133 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580051 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656178 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717894 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709537 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592608 s, 17.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8073 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4369 s, 10.0 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.71468 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9209 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.986291 s, 17.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721242 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707042 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11313 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894254 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092168 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986936 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121277 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608382 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126653 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201191 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186813 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210486 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236556 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148691 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214867 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221298 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999847 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257739 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338639 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404601 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122136 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482744 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423091 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565529 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090595 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963931 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905755 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612175 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642327 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063766 s, 16.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685735 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754171 s, 13.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7689 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6774 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.662443 s, 25.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139106 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114118 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20801 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175912 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167415 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208203 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262371 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183561 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359919 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258917 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319743 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495338 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419709 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511386 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362006 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.562897 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597044 s, 17.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181414 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941438 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129456 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790225 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123837 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114285 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146572 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807817 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110268 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192624 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104709 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109776 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280087 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428394 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138377 s, 7.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.49522 s, 14.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98211 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.691321 s, 24.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048754 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060582 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506767 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101134 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126054 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969435 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207137 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178127 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173411 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18741 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211016 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16619 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207279 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265932 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271981 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273392 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00242113 s, 433 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00276068 s, 380 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00227237 s, 461 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367305 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00527099 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00362033 s, 290 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00945551 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00259425 s, 404 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00226448 s, 463 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857522 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392425 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110595 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484416 s, 21.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195855 s, 53.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627715 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780202 s, 13.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8355 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98301 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5104 s, 10.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96102 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (439s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 16:51:57 (1789591917) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.0651 s, 15.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700781 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170908 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23514 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319055 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289245 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275332 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228322 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388659 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234418 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421007 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424222 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366679 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471499 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140392 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369491 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408643 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413729 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396042 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466395 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835603 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054902 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590838 s, 17.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612956 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791734 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512843 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479814 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488981 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117852 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595331 s, 17.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673433 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117516 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087909 s, 11.9 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.69483 s, 12.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98696 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (52s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 16:52:49 (1789591969) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.36@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.36@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.613003 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446243 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697141 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408503 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111315 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475317 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127943 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541348 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114475 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231414 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20142 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244161 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276304 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260839 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265004 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263715 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41546 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510936 s, 20.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362927 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659211 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801233 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410863 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441613 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518147 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689578 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597686 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664696 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587664 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658113 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696392 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686593 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958507 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806484 s, 13.0 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.70502 s, 15.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95032 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (49s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 16:53:38 (1789592018) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.36@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.587161 s, 28.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518019 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102139 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110013 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140477 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810538 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602404 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129308 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679984 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830758 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144678 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25677 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239637 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277294 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244428 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250975 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405949 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319171 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668429 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128011 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366958 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705815 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353278 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00237012 s, 442 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243255 s, 431 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047124 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548339 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555017 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309839 s, 33.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8aa484c95000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651873 s, 16.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439993 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619166 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569611 s, 18.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.36955 s, 16.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.28704 s, 19.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (46s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 16:54:24 (1789592064) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.810502 s, 20.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873553 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878017 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924681 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871416 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160834 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241596 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281928 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299591 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187036 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226662 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235828 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379726 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364893 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258113 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300515 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285203 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939923 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316158 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893482 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424665 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917549 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088267 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471524 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101865 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563651 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556875 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547062 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628664 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629932 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649001 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669598 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704135 s, 14.9 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.5502 s, 7.7 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0577 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.61773 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95938 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (102s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 16:56:06 (1789592166) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.983892 s, 17.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530332 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667929 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865872 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114004 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207021 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173522 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178814 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272064 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276219 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193581 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22274 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228918 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298761 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279965 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313272 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27969 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232973 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795806 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648642 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019212 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747098 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033587 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998619 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702172 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483002 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420622 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562272 s, 18.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468485 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494366 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506805 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897444 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600496 s, 17.5 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8641 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9696 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (76s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 16:57:22 (1789592242) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.614336 s, 27.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702817 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753363 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120621 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112406 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211066 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197856 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197198 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258688 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290697 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219671 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222981 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273982 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274772 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312374 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255095 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272863 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa484c95000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130211 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138304 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00357602 s, 293 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00907959 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726122 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08369 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128381 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668189 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11085 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107645 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100452 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1665 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168637 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126725 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493776 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175 s, 6.0 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.03962 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99941 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg136-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.635085 s, 26.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645512 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120948 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130371 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168588 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221444 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212051 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206031 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241266 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283192 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312052 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237194 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247145 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332541 s, 3.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27856 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336723 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305134 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100576 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188536 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069527 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808755 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256051 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525328 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765621 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035683 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377648 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395824 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500614 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513508 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050983 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531136 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659823 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070062 s, 15.0 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8911 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0013 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.139 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0175 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (148s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 16:59:50 (1789592390) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg136-client.virtnet /mnt/lustre (opts:) Stopping client oleg136-client.virtnet /mnt/lustre opts: Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg136-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg136-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg136-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg136-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg136-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg136-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.36,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg136-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.36,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg136-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.36,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70361/testresults/sanityn-zfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.36,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg136-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: kmemleak disabled oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: detected 4 online CPUs by sysfs oleg136-server: MODOPTS_LIBCFS= oleg136-server: Force libcfs to create 2 CPU partitions oleg136-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0001 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre Starting client oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre Started clients oleg136-client.virtnet: 192.168.201.136@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2 Starting client oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2 Started clients oleg136-client.virtnet: 192.168.201.136@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8aa485481800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8aa488d4d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8aa485481800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8aa488d4d000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.427702 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822155 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901921 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12878 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123963 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307867 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211732 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321766 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236256 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233049 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341524 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278964 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338396 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27737 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287334 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379364 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301754 s, 3.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8aa485481800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759109 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00512529 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252819 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554374 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338145 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00836178 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512421 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771275 s, 13.6 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff8aa485481800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00464485 s, 226 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284467 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120633 s, 86.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102632 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117025 s, 89.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148274 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172216 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00319781 s, 328 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.04791 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23121 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.572758 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115219 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128275 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539681 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796326 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17161 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180339 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150088 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23939 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247779 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21414 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122875 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229221 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286638 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205202 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255961 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438926 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561639 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060523 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476415 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103019 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550548 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504984 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061705 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549019 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472154 s, 22.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665824 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784801 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576464 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819656 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822085 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731131 s, 14.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.34346 s, 16.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90318 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (240s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 17:03:51 (1789592631) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg136-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (19s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 17:04:10 (1789592650) On MGS 192.168.201.136, active = nodemap.active=1 On MGS 192.168.201.136, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.586084 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846845 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105015 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107514 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602748 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142479 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909805 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135891 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248124 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151711 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226791 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203947 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276105 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280171 s, 3.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259563 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226151 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622619 s, 16.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa485481800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185216 s, 56.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325664 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397059 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036014 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350487 s, 29.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182571 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303402 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580518 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569694 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419314 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363088 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499659 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381171 s, 27.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395187 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00326968 s, 321 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483875 s, 21.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.47322 s, 14.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.44405 s, 19.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.201.136, TBF.id = On MGS 192.168.201.136, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.508381 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427661 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868524 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121326 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593839 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14364 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669467 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166041 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422472 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769494 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253019 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231179 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297105 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295418 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161604 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347513 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755792 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109632 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305609 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195212 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960554 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085204 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412169 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00889533 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379745 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202744 s, 51.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa485481800.lock_unused_count=9 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154226 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226603 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00607783 s, 173 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018666 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00268604 s, 390 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162479 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154008 s, 68.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.32296 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.36303 s, 44.4 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg136-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg136-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 On MGS 192.168.201.136, active = nodemap.active=1 On MGS 192.168.201.136, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.674939 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085507 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876588 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167787 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1757 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213839 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216735 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226627 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219644 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259497 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202653 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262514 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219881 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251149 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26972 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253696 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240399 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470222 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563301 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057581 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604437 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533414 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945863 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649204 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102609 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106105 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727839 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584323 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655209 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838844 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752252 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711311 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113167 s, 9.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.94475 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0705 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.136, TBF.id = On MGS 192.168.201.136, active = nodemap.active=0 PASS 77kj (180s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 17:07:10 (1789592830) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 17:07:20 (1789592840) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg136-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 4.02556 s, 260 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg136-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1629 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 17:08:19 (1789592899) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.684026 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610101 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114156 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114434 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699815 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240555 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427791 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276568 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269929 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179567 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151458 s, 6.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8aa485481800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17048 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37306 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.625947 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443823 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.536688 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814931 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294692 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303896 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.604387 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023985 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956852 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274475 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292459 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300612 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa485481800.lock_unused_count=13 ldlm.namespaces.lustre-OST0001-osc-ffff8aa485481800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332552 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341353 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342856 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190279 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343137 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239626 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14134 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347431 s, 3.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg136-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg136-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.34755 s, 12.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.55483 s, 18.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.96835 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619606 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14666 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22198 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869576 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120053 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895755 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124204 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324733 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157569 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230905 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340171 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493988 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257872 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257471 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351336 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287143 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa485481800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8aa485481800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311117 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446569 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012241 s, 85.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569693 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267869 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105131 s, 99.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00238891 s, 439 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056703 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00239125 s, 439 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104087 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346696 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135736 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220621 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191423 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102965 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433554 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8aa485481800.lock_unused_count=2 removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg136-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg136-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 17.1309 s, 6.1 MB/s Write runtime is 19 s, speed is 5.263157 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.68979 s, 18.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (106s) == sanityn test 77o: Changing rank should not panic ====== 17:10:06 (1789593006) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 17:10:18 (1789593018) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (124s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 17:12:22 (1789593142) mds.MDS.mdt.nrs_policies=tbf oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg136-client: oleg136-server: ssh exited with exit code 36 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 oleg136-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg136-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg136-client: oleg136-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (48s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 17:13:10 (1789593190) striped dir -i0 -c1 -H crush2 /mnt/lustre/d77r.sanityn mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 293951 kill md_thread xargs PASS 77r (62s) == sanityn test 77s: Check TBF LRU shrinker ============== 17:14:12 (1789593252) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (63s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 17:15:15 (1789593315) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 17:15:26 (1789593326) striped dir -i1 -c2 -H crush /mnt/lustre/d79.sanityn File: /mnt/lustre/d79.sanityn Size: 32768 Blocks: 28 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 162129637319639057 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 17:15:28.000000000 -0400 Modify: 2026-09-16 17:15:28.000000000 -0400 Change: 2026-09-16 17:15:28.000000000 -0400 Birth: 2026-09-16 17:15:28.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (23s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 17:15:48 (1789593348) total: 10 open/close in 0.34 seconds: 29.81 ops/second multiop /mnt/lustre2/d80a.sanityn/f80a.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 Finish migration, then checking.. PASS 80a (14s) == sanityn test 80b: Accessing directory during migration ========================================================== 17:16:02 (1789593362) start migration thread 300746 accessing the migrating directory for 1 minute... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete concurrent access 3 failures, 24 successes PASS 80b (135s) == sanityn test 81a: rename and stat under striped directory ========================================================== 17:18:18 (1789593498) stat: cannot statx '/mnt/lustre/d81a.sanityn/d0/0': No such file or directory File: /mnt/lustre/d81a.sanityn/d0/0 Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272381367660 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 17:18:20.000000000 -0400 Modify: 2026-09-16 17:18:20.000000000 -0400 Change: 2026-09-16 17:18:20.000000000 -0400 Birth: 2026-09-16 17:18:19.000000000 -0400 PASS 81a (8s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 17:18:25 (1789593505) - open/close 742 (time 1789593519.54 total 10.02 last 74.06) - open/close 1442 (time 1789593529.54 total 20.02 last 69.98) total: 2500 open/close in 29.97 seconds: 83.43 ops/second /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7851: 302447 Killed ( while true; do touch $DIR1/$tdir; done ) (wd: ~) PASS 81b (487s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 17:26:32 (1789593992) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 17:26:36 (1789593996) - open/close 707 (time 1789594012.80 total 10.01 last 70.65) - open/close 1380 (time 1789594022.80 total 20.01 last 67.26) - open/close 2196 (time 1789594032.81 total 30.02 last 81.55) total: 2500 open/close in 32.72 seconds: 76.41 ops/second - open/close 1058 (time 1789594050.07 total 10.00 last 105.77) - open/close 1855 (time 1789594060.08 total 20.01 last 79.68) total: 2500 open/close in 27.72 seconds: 90.17 ops/second snapshot_time 1789594221.708701764 secs.nsecs start_time 1789593999.603106080 secs.nsecs elapsed_time 222.105595684 secs.nsecs open 2505 samples [usecs] 114 23674 3497665 11812988687 close 2504 samples [usecs] 150 9306 853046 540863606 mknod 2501 samples [usecs] 407 23546 2939066 8862967638 unlink 2500 samples [usecs] 507 280379 7245839 190649974315 mkdir 2 samples [usecs] 1632 9364 10996 90347920 rmdir 2 samples [usecs] 4519 4733 9252 42822650 rename 2500 samples [usecs] 388 67797 20359875 384255377475 getattr 3766 samples [usecs] 1 53791 331439 3653975699 statfs 43 samples [usecs] 17 132 1172 44488 parallel_rename_file 2500 samples [usecs] 388 67797 20359875 384255377475 crossdir_rename 2500 samples [usecs] 388 67797 20359875 384255377475 readpage 30 samples [usecs] 4226 162156 928776 53318940842 snapshot_time 1789594221.708786338 secs.nsecs start_time 1789593999.603168029 secs.nsecs elapsed_time 222.105618309 secs.nsecs open 2504 samples [usecs] 95 13370 3029173 6230154411 close 2504 samples [usecs] 142 9453 889628 601008984 mknod 2500 samples [usecs] 445 11806 2537846 4395110298 unlink 2500 samples [usecs] 443 234517 7518633 200620937273 mkdir 2 samples [usecs] 1007 19322 20329 374353733 rmdir 2 samples [usecs] 27535 40632 68167 2409135649 rename 2500 samples [usecs] 383 49196 14072734 167886687656 getattr 3766 samples [usecs] 6 26264 222245 1082510283 statfs 43 samples [usecs] 16 165 1203 53573 parallel_rename_file 2500 samples [usecs] 383 49196 14072734 167886687656 crossdir_rename 2500 samples [usecs] 383 49196 14072734 167886687656 readpage 33 samples [usecs] 5809 186258 1172595 80891397003 PASS 81d (232s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 17:30:28 (1789594228) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.8447 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.8447 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 17:30:36 (1789594236) start pid 316485 to create/unlink striped directory start pid 316486 to stat striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6198: 316485 Terminated ( cd $DIR1; while true; do $LFS mkdir -i1 -c2 $tdir; rmdir $tdir; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6198: 316486 Terminated ( cd $DIR2; while true; do stat $tdir > /dev/null 2>&1; done ) PASS 83 (129s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 17:32:44 (1789594364) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 17:32:57 (1789594377) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789594380 Finishing test test1 at 1789594384 PASS 85 (13s) == sanityn test 90: open/create and unlink striped directory ========================================================== 17:33:10 (1789594390) start pid 321234 to open/create under striped directory start pid 321235 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6265: 321234 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; touch $tdir/f{0..3} > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6265: 321235 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 90 (188s) == sanityn test 91: chmod and unlink striped directory === 17:36:18 (1789594578) start pid 323480 to chmod striped directory start pid 323482 to unlink striped directory /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6302: 323480 Terminated ( cd $DIR1; while true; do $LFS mkdir -c$MDSCOUNT $tdir > /dev/null 2>&1; chmod go+w $tdir > /dev/null 2>&1; done ) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6302: 323482 Terminated ( cd $DIR2; while true; do rm -rf $tdir > /dev/null 2>&1; done ) PASS 91 (188s) == sanityn test 92: create remote directory under orphan directory ========================================================== 17:39:26 (1789594766) fail_loc=0x1408 mkdir: cannot create directory '/mnt/lustre2/d92.sanityn/dir': No such file or directory lfs setdirstripe: cannot resolve path '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory (2) lfs setdirstripe: '/mnt/lustre2/d92.sanityn/remote_dir' is not on a Lustre filesystem: No such file or directory (2) lfs setdirstripe: cannot create dir '/mnt/lustre2/d92.sanityn/remote_dir': No such file or directory fail_loc=0 PASS 92 (9s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 17:39:35 (1789594775) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 3569 0xdf1 0x2c0000400 0 3603 0xe13 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5405 0x151d 0x280000401 1 5369 0x14f9 0x2c0000401 rc1=0 and rc2=0 PASS 93 (22s) == sanityn test 94: signal vs CP callback race =========== 17:39:57 (1789594797) osc.lustre-OST0000-osc-ffff8aa485481800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8aa488d4d000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8aa485481800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8aa488d4d000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.178974 s, 45.8 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.93549 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 327796 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.115284 s, 35.5 kB/s osc.lustre-OST0000-osc-ffff8aa485481800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8aa488d4d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8aa485481800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8aa488d4d000.idle_timeout=debug PASS 94 (18s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 17:40:15 (1789594815) llite.lustre-ffff8aa485481800.fast_read=0 llite.lustre-ffff8aa488d4d000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0714362 s, 29.4 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8aa485481800.fast_read=1 llite.lustre-ffff8aa488d4d000.fast_read=1 PASS 95a (25s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 17:40:40 (1789594840) llite.lustre-ffff8aa485481800.fast_read=0 llite.lustre-ffff8aa488d4d000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00613614 s, 2.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8aa485481800.fast_read=1 llite.lustre-ffff8aa488d4d000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 17:40:54 (1789594854) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 17:40:58 (1789594858) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0128006 s, 320 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (9s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 17:41:07 (1789594867) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178712 s, 11.7 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (10s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 17:41:17 (1789594877) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.32738 s, 6.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (10s) == sanityn test 100e: DoM: read on open and file size ==== 17:41:27 (1789594887) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00408152 s, 2.9 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 17:41:35 (1789594895) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0122656 s, 334 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 17:41:43 (1789594903) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0144376 s, 284 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 17:41:51 (1789594911) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0295834 s, 138 kB/s PASS 101c (9s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 17:42:00 (1789594920) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 04 17 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0x1704:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote fh_bytes: 32 fh_type: 151 fh_data: d4 0b 00 40 02 00 00 00 1e 13 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd4:0x131e:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.remote/subdir fh_bytes: 32 fh_type: 151 fh_data: a1 13 00 00 02 00 00 00 05 17 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x2000013a1:0x1705:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe fh_bytes: 32 fh_type: 151 fh_data: d4 0b 00 40 02 00 00 00 1f 13 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd4:0x131f:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! file: /mnt/lustre/d102.sanityn.stripe/subdir fh_bytes: 32 fh_type: 151 fh_data: d4 0b 00 40 02 00 00 00 20 13 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x240000bd4:0x1320:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! PASS 102 (10s) == sanityn test 103: Test size correctness with lockahead ========================================================== 17:42:10 (1789594930) striped dir -i1 -c2 -H crush /mnt/lustre/d103.sanityn fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789594935 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789594938 Finishing test test23 at 1789594938 PASS 103 (16s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 17:42:26 (1789594946) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 17:42:30 (1789594950) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d105.sanityn fail_loc=0x416 PASS 105 (25s) == sanityn test 106a: Verify the btime via statx() ======= 17:42:55 (1789594975) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 17:42:59 (1789594979) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0447513 s, 22.9 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381372168 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 17:43:01.000000000 -0400 Modify: 2026-09-16 17:43:01.000000000 -0400 Change: 2026-09-16 17:43:01.000000000 -0400 Birth: 2026-09-16 17:43:01.000000000 -0400 /mnt/lustre/f106b.sanityn 144115272381372168 -rw-r--r-- 1 0 0 1789594981 1789594981 1789594981 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381372168 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 17:43:01.000000000 -0400 Modify: 2026-09-16 17:43:01.000000000 -0400 Change: 2026-09-16 17:43:01.000000000 -0400 Birth: 2026-09-16 17:43:01.000000000 -0400 1789594981 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 17:43:08 (1789594988) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 17:43:15 (1789594995) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.403592 s, 26.0 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5375 0x14ff 0x2c0000401 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 17:43:24 (1789595004) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.390083 s, 26.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5412 0x1524 0x280000401 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.8447 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 17:43:38 (1789595018) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00474574 s, 2.1 MB/s llite.lustre-ffff8aa485481800.hybrid_io=0 llite.lustre-ffff8aa488d4d000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.84107 s, 1.7 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 17:43:50 (1789595030) 192.168.201.136@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg136-client.virtnet /mnt/lustre (opts:) 192.168.201.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2 Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre 192.168.201.136@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg136-client.virtnet /mnt/lustre (opts:) Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg136-server' oleg136-server: oleg136-server.virtnet: executing load_modules_local oleg136-server: kmemleak disabled oleg136-server: Loading modules from /home/green/git/lustre-release/lustre oleg136-server: detected 4 online CPUs by sysfs oleg136-server: MODOPTS_LIBCFS= oleg136-server: Force libcfs to create 2 CPU partitions oleg136-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2 PASS 109 (56s) == sanityn test 110: do not grant another lock on resend ========================================================== 17:44:46 (1789595086) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115272431697923 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-16 17:44:48.000000000 -0400 Modify: 2026-09-16 17:44:48.000000000 -0400 Change: 2026-09-16 17:44:48.000000000 -0400 Birth: 2026-09-16 17:44:48.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg136-client.virtnet oleg136-client.virtnet PASS 110 (70s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 17:45:56 (1789595156) fail_loc=0x8000018a /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 7188: 342230 Terminated $MULTIOP $DIR2/$tdir/mdt0dir/foodir/file2 Ow4096_c PASS 111 (11s) == sanityn test 112: update max-inherit in default LMV === 17:46:07 (1789595167) lmv.lustre-clilmv-ffff8aa490955000.qos_threshold_rr=90 lmv.lustre-clilmv-ffff8aa491d0c000.qos_threshold_rr=90 PASS 112 (13s) == sanityn test 113: check servers of specified fs ======= 17:46:20 (1789595180) lustre-OST0000-osc-ffff8aa490955000 active. lustre-OST0001-osc-ffff8aa490955000 active. lustre-OST0000-osc-ffff8aa490955000 active. lustre-OST0001-osc-ffff8aa490955000 active. MGC192.168.201.136@tcp active. PASS 113 (6s) == sanityn test 114: implicit default LMV inherit ======== 17:46:26 (1789595186) /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1 lmv_stripe_count: 2 lmv_stripe_offset: 2 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 2 lmv_stripe_offset: 1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 /mnt/lustre2/d114.sanityn/l1/l2/l3 lmv_stripe_count: 1 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 1 lmv_max_inherit_rr: 0 PASS 114 (21s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 17:46:47 (1789595207) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 17:46:50 (1789595210) llite.lustre-ffff8aa490955000.intent_mkdir=1 llite.lustre-ffff8aa491d0c000.intent_mkdir=1 MD layout /mnt/lustre/d116.sanityn: lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x2000013a0:0xed:0x0] 1 [0x240000bd2:0xec:0x0] mkdir /mnt/lustre/d116.sanityn/tdir0 setdirstripe -D -i1 /mnt/lustre2/d116.sanityn/tdir0 mkdir /mnt/lustre/d116.sanityn/tdir0/tdir11 lmv_stripe_count: 0 lmv_stripe_offset: 0 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 1 lmv_hash_type: none PASS 116 (9s) == sanityn test 121: trunc append race =================== 17:46:59 (1789595219) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.025643 s, 799 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0733755 s, 279 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 17:47:09 (1789595229) PASS 122 (19s) == sanityn test 200: service remains healthy while able to process request ========================================================== 17:47:28 (1789595248) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 striped dir -i0 -c2 -H crush /mnt/lustre/d200.sanityn fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.149 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.297 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1914 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2552 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2321 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1945 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2269 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1293 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2036 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2524 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2719 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3106 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2418 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.323 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3504 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4326 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.464 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5478 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8aa490955000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8aa490955000.ost_server_uuid in FULL state after 0 sec PASS 200 (83s) cleanup: ====================================================== == sanityn test complete, duration 12062 sec ============= 17:48:54 (1789595334) === sanityn: start cleanup 17:48:56 (1789595336) === Stopping clients: oleg136-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg136-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 17:53:24 (1789595604) ===