-----============= acceptance-small: sanityn ============----- Wed Jun 3 20:36:03 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg339-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 20:36:13 (1780533373) === Starting client oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Mount client oleg339-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Started clients oleg339-client.virtnet: 192.168.203.139@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg339-client.virtnet: executing check_config_client /mnt/lustre oleg339-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg339-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff925845219000.idle_timeout=debug osc.lustre-OST0000-osc-ffff92584899c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff925845219000.idle_timeout=debug osc.lustre-OST0001-osc-ffff92584899c000.idle_timeout=debug disable quota as required oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 20:36:25 (1780533385) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014602 s, 71.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7577] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 20:36:26 (1780533386) PASS 0 (3s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 20:36:29 (1780533389) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (3s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 20:36:32 (1780533392) -rw-r--r-- 1 root root 0 Jun 3 20:36 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 20:36:34 (1780533394) -rw-r--r-- 1 root root 0 Jun 3 20:36 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (3s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 20:36:37 (1780533397) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (3s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 20:36:40 (1780533400) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 20:36:42 (1780533402) total 1 -rwxrwxrwx 1 root root 0 Jun 3 20:36 f2a -rwxrwxrwx 1 root root 0 Jun 3 20:36 f2b total 1 -rwxrwxrwx 1 root root 0 Jun 3 20:36 f2a -rwxrwxrwx 1 root root 0 Jun 3 20:36 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00653057 s, 78.4 kB/s PASS 2e (3s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 20:36:45 (1780533405) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 20:36:46 (1780533406) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146797 s, 14.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (3s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 20:36:49 (1780533409) PASS 3 (3s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 20:36:52 (1780533412) PASS 4 (4s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 20:36:56 (1780533416) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (3s) == sanityn test 6: remove of open file on other node ============================================================================ 20:36:59 (1780533419) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (3s) == sanityn test 7: remove of open directory on other node ======================================================================= 20:37:02 (1780533422) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (3s) == sanityn test 8: remove of open special file on other node ==================================================================== 20:37:05 (1780533425) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (4s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 20:37:09 (1780533429) PASS 9a (3s) == sanityn test 9b: append to striped sparse file ======== 20:37:12 (1780533432) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232213 s, 45.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.029702 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (3s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 20:37:15 (1780533435) 1+0 records in 1+0 records out 1 byte copied, 0.00243982 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00594671 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00690848 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00897695 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0078934 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00699305 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00680072 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00897112 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00815169 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00659842 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00545014 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00665192 s, 0.2 kB/s PASS 10a (4s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 20:37:19 (1780533439) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00707661 s, 434 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0212805 s, 192 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000250698 s, 12.3 MB/s PASS 10b (3s) == sanityn test 11: execution of file opened for write should return error ============================================================== 20:37:22 (1780533442) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (4s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 20:37:26 (1780533446) warning: '-runtime' deprecated, use '-t runtime' instead - link 743 (time 1780533457.28 total 10.01 last 74.19) - link 1735 (time 1780533467.29 total 20.02 last 99.13) - link 3034 (time 1780533477.29 total 30.03 last 129.85) - link 3946 (time 1780533487.31 total 40.04 last 91.09) - link 4824 (time 1780533497.31 total 50.04 last 87.75) - link 5825 (time 1780533507.31 total 60.04 last 100.10) - link 6781 (time 1780533517.32 total 70.05 last 95.54) - link 7769 (time 1780533527.33 total 80.06 last 98.68) - link 8699 (time 1780533537.33 total 90.06 last 92.98) total: 9615 link in 100.00 seconds: 96.15 ops/second using seed 707067581 running for 100 seconds - stat 9575 (time 1780533459 ; total 11 ; last 11) - stat 10000 (time 1780533459 ; total 11 ; last 0) - stat 20000 (time 1780533469 ; total 21 ; last 10) - stat 30000 (time 1780533478 ; total 30 ; last 9) - stat 39776 (time 1780533489 ; total 41 ; last 11) - stat 40000 (time 1780533489 ; total 41 ; last 0) - stat 49759 (time 1780533500 ; total 52 ; last 11) - stat 50000 (time 1780533500 ; total 52 ; last 0) - stat 54575 (time 1780533511 ; total 63 ; last 11) - stat 59421 (time 1780533522 ; total 74 ; last 11) - stat 60000 (time 1780533523 ; total 75 ; last 1) - stat 64483 (time 1780533534 ; total 86 ; last 11) - stat 69355 (time 1780533545 ; total 97 ; last 11) - stat 70000 (time 1780533546 ; total 98 ; last 1) total: 70941 stats in 100 seconds: 709.409973 stats/second - unlinked 0 (time 1780533498 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile9615) error: No such file or directory total: 9614 unlinks in 103 seconds: 93.339806 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21347) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21349) - No such process PASS 12 (157s) == sanityn test 13: test directory page revocation ======= 20:40:04 (1780533604) f13 f13 PASS 13 (4s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 20:40:08 (1780533608) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5003: 22537 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 20:40:11 (1780533611) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5003: 23115 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (3s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 20:40:14 (1780533614) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23693 Terminated $DIR1/$tdir/sleep 60 PASS 14b (3s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 20:40:17 (1780533617) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24275 Terminated $DIR1/$tdir/sleep 60 PASS 14c (3s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 20:40:20 (1780533620) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24856 Terminated $DIR1/$tdir/sleep 60 PASS 14d (3s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 20:40:23 (1780533623) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7527424 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7527424KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (34s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 20:40:57 (1780533657) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.202833 s, 41.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.167658 s, 50.0 MB/s Chance of close/open is 1 in 50 Seed set to 3659 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[0] 1780533659.365201 mapwrite 0x623363 thru 0x625bb4 (0x2852 bytes) 000004[1] 1780533659.439630 mapread 0x327228 thru 0x334aa2 (0xd87b bytes) 000005[1] 1780533659.491651 mapread 0x12c97e thru 0x136bd1 (0xa254 bytes) 000006[0] 1780533659.519317 read 0x2d8915 thru 0x2e3862 (0xaf4e bytes) 000008[1] 1780533659.538129 mapread 0x5fdc4f thru 0x604e01 (0x71b3 bytes) 000009[0] 1780533659.565528 write 0x3e14ee thru 0x3e9da5 (0x88b8 bytes) truncating to largest ever: 0x788ef 000011[1] 1780533659.586374 trunc from 0x625bb5 to 0x0788ee (0x5ad2c6 bytes) truncating to largest ever: 0x5b4f8c 000012[0] 1780533659.618950 trunc from 0x0788ef to 0x5b4f8b (0x53c69d bytes) 000013[1] 1780533659.645699 write 0x89eca0 thru 0x89f8ae (0xc0f bytes) 000016[0] 1780533659.695274 read 0x60cb40 thru 0x61b964 (0xee25 bytes) 000021[1] 1780533659.847827 read 0x703de6 thru 0x7123ac (0xe5c7 bytes) 000023[0] 1780533659.909794 write 0x763aaf thru 0x770b8d (0xd0df bytes) 000024[0] 1780533659.924010 read 0x85caa2 thru 0x867bfb (0xb15a bytes) 000026[1] 1780533659.948910 mapread 0x3e0f0a thru 0x3ef1e7 (0xe2de bytes) 000028[1] 1780533659.988939 trunc from 0x89f8af to 0x007e2e (0x897a80 bytes) 000032[1] 1780533660.048434 mapread 0x006aa9 thru 0x007e2e (0x1386 bytes) 000034[0] 1780533660.070632 mapread 0x002a9f thru 0x007e2e (0x5390 bytes) 000035[1] 1780533660.097287 mapread 0x007345 thru 0x007e2e (0xaea bytes) truncating to largest ever: 0x5fc1dc 000036[1] 1780533660.120535 trunc from 0x007e2f to 0x5fc1db (0x5f43ad bytes) 000037[1] 1780533660.162937 read 0x292b7e thru 0x298bee (0x6071 bytes) 000040[0] 1780533660.196337 mapwrite 0x47efaa thru 0x48a371 (0xb3c8 bytes) 000042[0] 1780533660.278408 read 0x467a1c thru 0x469f17 (0x24fc bytes) 000043[0] 1780533660.292048 mapread 0x182ee4 thru 0x183781 (0x89e bytes) 000044[0] 1780533660.313204 mapwrite 0x2d34dd thru 0x2dfb86 (0xc6aa bytes) 000047[1] 1780533660.417030 write 0x67f664 thru 0x68ba4e (0xc3eb bytes) 000048[1] 1780533660.438851 trunc from 0x68ba4f to 0x3123a0 (0x3796ae bytes) 000050[1] 1780533660.484286 trunc from 0x3123a1 to 0x2f4fda (0x1d3c6 bytes) 000051[1] 1780533660.500502 trunc from 0x2f4fdb to 0x106f84 (0x1ee056 bytes) 000054[1] 1780533660.539154 mapwrite 0x70ffbe thru 0x711c84 (0x1cc7 bytes) 000055[0] 1780533660.627160 write 0x74cde7 thru 0x758db5 (0xbfcf bytes) 000056[0] 1780533660.647844 write 0x928541 thru 0x93252d (0x9fed bytes) truncating to largest ever: 0x7ae2cb 000060[0] 1780533660.682790 trunc from 0x93252e to 0x7ae2ca (0x184263 bytes) 000063[1] 1780533660.717691 mapread 0x26a9c3 thru 0x26dde9 (0x3427 bytes) 000064[0] 1780533660.749477 mapwrite 0x2f8b40 thru 0x3021a9 (0x966a bytes) 000065[1] 1780533660.804421 read 0x1ee3d6 thru 0x1fc446 (0xe071 bytes) 000066[1] 1780533660.861826 trunc from 0x7ae2cb to 0x305a25 (0x4a88a5 bytes) 000069[0] 1780533660.907884 write 0x867f66 thru 0x871521 (0x95bc bytes) 000071[1] 1780533660.938524 read 0x8453c0 thru 0x845795 (0x3d6 bytes) 000073[1] 1780533660.977410 write 0x71c48a thru 0x723180 (0x6cf7 bytes) 000074[0] 1780533661.002128 read 0x6fe53e thru 0x70b971 (0xd434 bytes) 000075[0] 1780533661.024464 mapwrite 0x4e1c54 thru 0x4e5a57 (0x3e04 bytes) 000078[0] 1780533661.126600 read 0x527a6e thru 0x52e2b9 (0x684c bytes) 000079[1] 1780533661.154697 trunc from 0x871522 to 0x5c6ac0 (0x2aaa61 bytes) 000080[1] 1780533661.192284 mapread 0x05b777 thru 0x06698f (0xb219 bytes) 000083[1] 1780533661.242390 mapwrite 0x489127 thru 0x495a70 (0xc94a bytes) 000084[0] 1780533661.311225 mapwrite 0x83c7a3 thru 0x83e9d8 (0x2236 bytes) 000085[1] 1780533661.388304 read 0x3d6f2c thru 0x3e2072 (0xb147 bytes) 000086[1] 1780533661.423410 mapread 0x6bed61 thru 0x6c26ca (0x396a bytes) 000088[1] 1780533661.450059 trunc from 0x83e9d9 to 0x35d8cd (0x4e110b bytes) 000089[1] 1780533661.477359 mapread 0x3227b7 thru 0x32fd00 (0xd54a bytes) 000095[0] 1780533661.587833 read 0x2991bc thru 0x29c0ae (0x2ef3 bytes) 000097[1] 1780533661.659855 trunc from 0x35d8ce to 0x18cb96 (0x1d0d37 bytes) 000098[0] 1780533661.699286 mapwrite 0x0113fd thru 0x016b9a (0x579e bytes) 000101[1] 1780533661.827149 trunc from 0x18cb97 to 0x10a36e (0x82828 bytes) 000102[1] 1780533661.868482 read 0x000fac thru 0x002314 (0x1369 bytes) 000104[0] 1780533661.889319 write 0x9940b9 thru 0x99de42 (0x9d8a bytes) 000106[1] 1780533661.920164 mapread 0x94f180 thru 0x95ba45 (0xc8c6 bytes) 000107[0] 1780533661.994696 read 0x272b1f thru 0x27f5d1 (0xcab3 bytes) 000108[1] 1780533662.006090 mapwrite 0x1d92fd thru 0x1e2f03 (0x9c07 bytes) 000110[1] 1780533662.102872 read 0x4bd171 thru 0x4c3333 (0x61c3 bytes) 000113[1] 1780533662.129060 mapwrite 0x243bf1 thru 0x24a158 (0x6568 bytes) 000115[0] 1780533662.176989 read 0x8d9763 thru 0x8df53c (0x5dda bytes) 000116[0] 1780533662.206965 mapwrite 0x2c3ef0 thru 0x2cdb10 (0x9c21 bytes) 000118[1] 1780533662.288401 mapread 0x3b4e85 thru 0x3c3ff5 (0xf171 bytes) 000119[1] 1780533662.325069 mapwrite 0x4f0da6 thru 0x4f460f (0x386a bytes) 000121[0] 1780533662.368503 write 0x444d55 thru 0x44f978 (0xac24 bytes) 000124[0] 1780533662.403317 mapwrite 0x3a4d78 thru 0x3b4398 (0xf621 bytes) 000125[1] 1780533662.461232 mapread 0x16bed5 thru 0x173471 (0x759d bytes) 000126[0] 1780533662.493472 mapread 0x2316ca thru 0x234279 (0x2bb0 bytes) 000128[0] 1780533662.520384 read 0x64d7d9 thru 0x6536ef (0x5f17 bytes) 000129[0] 1780533662.552834 read 0x2b288e thru 0x2bf295 (0xca08 bytes) 000130[1] 1780533662.560473 mapread 0x00cfa7 thru 0x012a58 (0x5ab2 bytes) 000131[0] 1780533662.587041 write 0x2f908d thru 0x3083d3 (0xf347 bytes) 000136[1] 1780533662.620458 mapwrite 0x48e4b8 thru 0x493a71 (0x55ba bytes) 000140[1] 1780533662.732536 read 0x71eb04 thru 0x72c0fb (0xd5f8 bytes) 000144[1] 1780533662.834854 trunc from 0x99de43 to 0x648328 (0x355b1a bytes) 000145[1] 1780533662.870496 trunc from 0x648329 to 0x623fba (0x2436e bytes) 000146[0] 1780533662.880320 mapwrite 0x2b17ad thru 0x2c15a7 (0xfdfb bytes) 000148[1] 1780533662.953700 write 0x804a84 thru 0x811637 (0xcbb4 bytes) 000151[0] 1780533662.976309 write 0x931b80 thru 0x93e665 (0xcae6 bytes) 000152[0] 1780533663.002765 read 0x8483df thru 0x85117f (0x8da1 bytes) 000153[1] 1780533663.015436 mapwrite 0x59d894 thru 0x5a6412 (0x8b7f bytes) 000154[1] 1780533663.068091 mapwrite 0x93ae6f thru 0x946b95 (0xbd27 bytes) 000157[1] 1780533663.158244 mapwrite 0x0d0398 thru 0x0d1859 (0x14c2 bytes) 000162[0] 1780533663.229907 mapwrite 0x9135c7 thru 0x918920 (0x535a bytes) 000164[1] 1780533663.306027 write 0x2955cb thru 0x2a1149 (0xbb7f bytes) 000165[1] 1780533663.322251 read 0x1d506e thru 0x1da9ff (0x5992 bytes) 000167[1] 1780533663.350026 write 0x71a7cc thru 0x720da0 (0x65d5 bytes) 000168[0] 1780533663.366851 trunc from 0x946b96 to 0x47c6b4 (0x4ca4e1 bytes) 000174[1] 1780533663.441329 write 0x4a4438 thru 0x4aa4f3 (0x60bc bytes) 000176[1] 1780533663.464593 write 0x765884 thru 0x7714ab (0xbc28 bytes) 000178[0] 1780533663.484488 write 0x96184c thru 0x9701e1 (0xe996 bytes) 000180[0] 1780533663.506389 read 0x29aac9 thru 0x29feee (0x5426 bytes) 000181[0] 1780533663.523976 read 0x74155c thru 0x743d5a (0x27ff bytes) 000183[0] 1780533663.551945 mapwrite 0x8e2c10 thru 0x8ef950 (0xcd41 bytes) 000187[0] 1780533663.602098 trunc from 0x9701e2 to 0x352dc9 (0x61d418 bytes) 000188[0] 1780533663.626223 trunc from 0x352dca to 0x6ddf14 (0x38b14b bytes) 000189[0] 1780533663.635600 read 0x416164 thru 0x41872c (0x25c9 bytes) 000190[1] 1780533663.653712 mapwrite 0x9fdb73 thru 0x9fffff (0x248d bytes) 000192[1] 1780533663.742675 write 0x9440ba thru 0x94f143 (0xb08a bytes) 000194[1] 1780533663.771492 mapwrite 0x18e8da thru 0x19c20a (0xd931 bytes) 000195[0] 1780533663.853014 trunc from 0xa00000 to 0x767d07 (0x2982f8 bytes) 000200[1] 1780533663.917364 mapread 0x4887ec thru 0x48d27b (0x4a90 bytes) 000202[1] 1780533663.966934 mapwrite 0x8161aa thru 0x81ee51 (0x8ca8 bytes) 000205[0] 1780533664.043490 read 0x608764 thru 0x60fe66 (0x7703 bytes) 000206[0] 1780533664.061921 mapread 0x353370 thru 0x355609 (0x229a bytes) 000208[0] 1780533664.086156 read 0x01b8d6 thru 0x02a638 (0xed63 bytes) 000211[1] 1780533664.097510 mapwrite 0x083ab3 thru 0x086f55 (0x34a3 bytes) 000217[0] 1780533664.195282 mapwrite 0x7651fa thru 0x76d718 (0x851f bytes) 000219[1] 1780533664.269944 mapread 0x0ce69d thru 0x0d5d13 (0x7677 bytes) 000221[1] 1780533664.315460 write 0x68f82d thru 0x69121c (0x19f0 bytes) 000222[0] 1780533664.349655 write 0x52f3a8 thru 0x53804d (0x8ca6 bytes) 000223[0] 1780533664.372872 mapwrite 0x8a7c91 thru 0x8b0855 (0x8bc5 bytes) 000224[1] 1780533664.438193 trunc from 0x8b0856 to 0x7523a6 (0x15e4af bytes) 000225[1] 1780533664.480365 write 0x75ff9c thru 0x76a331 (0xa396 bytes) 000226[1] 1780533664.485350 write 0x0b75f9 thru 0x0c101b (0x9a23 bytes) 000228[1] 1780533664.519368 trunc from 0x76a332 to 0x058821 (0x711b10 bytes) 000229[0] 1780533664.532792 trunc from 0x058822 to 0x76809e (0x70f87d bytes) 000235[0] 1780533664.561390 trunc from 0x76809f to 0x66495f (0x10373f bytes) 000236[0] 1780533664.570009 mapwrite 0x15c5b9 thru 0x160f00 (0x4948 bytes) 000238[1] 1780533664.630872 mapread 0x388bff thru 0x393450 (0xa852 bytes) 000239[1] 1780533664.674900 mapread 0x37cf6b thru 0x38ad33 (0xddc9 bytes) 000241[1] 1780533664.711162 read 0x021cbd thru 0x02f551 (0xd895 bytes) 000242[0] 1780533664.718428 read 0x3051f1 thru 0x30817f (0x2f8f bytes) 000243[0] 1780533664.723479 mapwrite 0x965122 thru 0x968263 (0x3142 bytes) 000245[0] 1780533664.816985 mapwrite 0x88fdb3 thru 0x8919ac (0x1bfa bytes) 000246[0] 1780533664.856059 read 0x5a6cfb thru 0x5b3c2a (0xcf30 bytes) truncating to largest ever: 0x852403 000248[1] 1780533664.876260 trunc from 0x968264 to 0x852402 (0x115e61 bytes) 000250[1] 1780533664.925546 mapwrite 0x5736c7 thru 0x57cc10 (0x954a bytes) 000252[0] 1780533664.969986 mapread 0x4735df thru 0x482d30 (0xf752 bytes) 000254[1] 1780533665.026716 write 0x0e5f9c thru 0x0ed309 (0x736e bytes) 000256[0] 1780533665.052834 mapwrite 0x98886d thru 0x98ada1 (0x2535 bytes) 000260[0] 1780533665.151809 mapread 0x876ed9 thru 0x8814a5 (0xa5cd bytes) 000261[0] 1780533665.182848 write 0x0c2c57 thru 0x0c62aa (0x3654 bytes) 000262[0] 1780533665.189258 write 0x77be58 thru 0x787d56 (0xbeff bytes) 000264[1] 1780533665.207024 mapread 0x61b83d thru 0x61e8d8 (0x309c bytes) 000268[0] 1780533665.241870 write 0x98a6d6 thru 0x999741 (0xf06c bytes) 000271[1] 1780533665.252837 write 0x959ef8 thru 0x95ef5f (0x5068 bytes) 000272[0] 1780533665.278905 trunc from 0x999742 to 0x3b7179 (0x5e25c8 bytes) 000273[1] 1780533665.303320 trunc from 0x3b717a to 0x6f5cb3 (0x33eb3a bytes) 000274[1] 1780533665.334971 mapwrite 0x1bc57d thru 0x1bef7e (0x2a02 bytes) 000280[0] 1780533665.420249 write 0x33d6bb thru 0x34c4d0 (0xee16 bytes) 000281[1] 1780533665.439524 read 0x482d67 thru 0x484b63 (0x1dfd bytes) 000284[1] 1780533665.481286 write 0x0cedb3 thru 0x0cee9f (0xed bytes) 000285[1] 1780533665.502223 read 0x163a0f thru 0x168b8a (0x517c bytes) 000286[1] 1780533665.516274 read 0x3091bd thru 0x30b5f3 (0x2437 bytes) 000288[0] 1780533665.532046 mapwrite 0x8471db thru 0x856db3 (0xfbd9 bytes) 000290[1] 1780533665.627323 read 0x7120f9 thru 0x714196 (0x209e bytes) 000291[0] 1780533665.643811 write 0x17660f thru 0x182a22 (0xc414 bytes) 000292[1] 1780533665.656887 read 0x0df1d2 thru 0x0e4ac9 (0x58f8 bytes) 000296[1] 1780533665.682773 trunc from 0x856db4 to 0x333bd4 (0x5231df bytes) 000300[1] 1780533665.740281 trunc from 0x333bd5 to 0x40ad89 (0xd71b5 bytes) 000301[0] 1780533665.754950 write 0x71489d thru 0x71e71f (0x9e83 bytes) 000302[1] 1780533665.776361 write 0x6ba5cc thru 0x6c729e (0xccd3 bytes) 000303[0] 1780533665.792342 mapwrite 0x98a92f thru 0x99a303 (0xf9d5 bytes) 000304[1] 1780533665.871010 mapread 0x0cab33 thru 0x0d1166 (0x6634 bytes) 000305[1] 1780533665.899512 write 0x43a95d thru 0x43c1cd (0x1871 bytes) 000306[0] 1780533665.912706 trunc from 0x99a304 to 0x017027 (0x9832dc bytes) 000311[1] 1780533665.972496 mapread 0x004cee thru 0x01199a (0xccad bytes) 000315[0] 1780533666.039334 read 0x0105d2 thru 0x010cfe (0x72d bytes) 000316[0] 1780533666.044469 mapread 0x003fe5 thru 0x00760d (0x3629 bytes) 000317[0] 1780533666.057885 mapwrite 0x1665db thru 0x171155 (0xab7b bytes) 000318[0] 1780533666.120594 mapread 0x05d520 thru 0x06b065 (0xdb46 bytes) 000320[0] 1780533666.149351 read 0x0a9d99 thru 0x0ac193 (0x23fb bytes) 000321[1] 1780533666.159684 trunc from 0x171156 to 0x46e3f2 (0x2fd29d bytes) 000323[0] 1780533666.204228 mapwrite 0x2961e0 thru 0x29b090 (0x4eb1 bytes) 000324[0] 1780533666.273937 mapread 0x22956c thru 0x238e56 (0xf8eb bytes) 000325[1] 1780533666.288593 trunc from 0x46e3f3 to 0x2af39b (0x1bf057 bytes) 000326[0] 1780533666.307191 read 0x1b7e65 thru 0x1c7558 (0xf6f4 bytes) 000327[0] 1780533666.319776 mapread 0x1e7fb0 thru 0x1f0ecd (0x8f1e bytes) 000329[1] 1780533666.363022 mapwrite 0x1de8da thru 0x1e1173 (0x289a bytes) 000331[1] 1780533666.426847 trunc from 0x2af39c to 0x008891 (0x2a6b0a bytes) 000332[1] 1780533666.437576 read 0x003a89 thru 0x008891 (0x4e09 bytes) 000333[1] 1780533666.463527 mapread 0x0030dd thru 0x008891 (0x57b5 bytes) 000334[0] 1780533666.464509 read 0x003580 thru 0x008891 (0x5312 bytes) 000336[0] 1780533666.489350 mapread 0x005fd1 thru 0x008891 (0x28c1 bytes) 000338[1] 1780533666.500248 mapwrite 0x2524cb thru 0x256173 (0x3ca9 bytes) 000342[1] 1780533666.597282 trunc from 0x256174 to 0x7cf9d2 (0x57985f bytes) 000343[0] 1780533666.622387 write 0x2336c7 thru 0x242355 (0xec8f bytes) 000344[0] 1780533666.672177 mapwrite 0x9f4851 thru 0x9fe3e8 (0x9b98 bytes) 000345[0] 1780533666.755232 write 0x29f12c thru 0x2a1dd7 (0x2cac bytes) 000346[0] 1780533666.769033 read 0x406c1f thru 0x4109e7 (0x9dc9 bytes) 000347[0] 1780533666.777706 mapwrite 0x27b818 thru 0x280cb5 (0x549e bytes) 000348[1] 1780533666.805054 trunc from 0x9fe3e9 to 0x1fb5e0 (0x802e08 bytes) 000349[0] 1780533666.836611 read 0x0fb932 thru 0x10210b (0x67da bytes) 000350[1] 1780533666.850460 mapread 0x01915a thru 0x027c1a (0xeac1 bytes) 000351[1] 1780533666.897980 write 0x3a0647 thru 0x3a39ce (0x3388 bytes) 000353[0] 1780533666.911891 read 0x2e90be thru 0x2f3576 (0xa4b9 bytes) 000354[1] 1780533666.944013 write 0x4b64e3 thru 0x4bf5eb (0x9109 bytes) 000355[1] 1780533666.957238 mapwrite 0x2c2bbb thru 0x2cd735 (0xab7b bytes) 000357[1] 1780533667.028254 trunc from 0x4bf5ec to 0x27a36d (0x24527e bytes) 000358[1] 1780533667.039922 read 0x1728de thru 0x17cb7b (0xa29e bytes) 000359[1] 1780533667.076093 write 0x7b31b5 thru 0x7b4a07 (0x1853 bytes) 000363[1] 1780533667.094556 read 0x6c5011 thru 0x6c7d76 (0x2d66 bytes) 000364[1] 1780533667.107128 write 0x83f31d thru 0x84099a (0x167e bytes) 000366[1] 1780533667.126512 mapwrite 0x279ef8 thru 0x282e56 (0x8f5f bytes) 000367[1] 1780533667.166148 write 0x679274 thru 0x682738 (0x94c5 bytes) 000368[0] 1780533667.184244 mapwrite 0x2e1e5e thru 0x2e68a9 (0x4a4c bytes) 000370[0] 1780533667.261039 mapwrite 0x0a5488 thru 0x0af661 (0xa1da bytes) 000371[1] 1780533667.335448 mapread 0x75520d thru 0x75aa6b (0x585f bytes) 000373[0] 1780533667.368591 read 0x5c451f thru 0x5c9985 (0x5467 bytes) 000376[1] 1780533667.412104 mapread 0x5b60fc thru 0x5c0136 (0xa03b bytes) 000377[1] 1780533667.434560 mapread 0x0013d0 thru 0x0072d8 (0x5f09 bytes) 000379[0] 1780533667.464531 trunc from 0x84099b to 0x5afa0e (0x290f8c bytes) 000382[0] 1780533667.509719 mapread 0x482bf3 thru 0x4857f2 (0x2c00 bytes) 000384[0] 1780533667.535271 mapread 0x0ba56b thru 0x0c018d (0x5c23 bytes) 000386[1] 1780533667.548314 mapwrite 0x5591f1 thru 0x566918 (0xd728 bytes) 000388[0] 1780533667.612265 read 0x2b2737 thru 0x2b762b (0x4ef5 bytes) 000390[1] 1780533667.631128 trunc from 0x5afa0f to 0x3dfaf7 (0x1cff17 bytes) 000392[1] 1780533667.657448 write 0x95e3f5 thru 0x96b448 (0xd054 bytes) 000395[1] 1780533667.661455 write 0x94b53b thru 0x959ec1 (0xe987 bytes) 000397[1] 1780533667.675323 mapwrite 0x435153 thru 0x44006e (0xaf1c bytes) 000398[1] 1780533667.730560 mapwrite 0x91ef25 thru 0x9238b9 (0x4995 bytes) 000399[1] 1780533667.776530 trunc from 0x96b449 to 0x2a618e (0x6c52ba bytes) 000400[1] 1780533667.790907 mapread 0x281990 thru 0x28f491 (0xdb02 bytes) 000401[1] 1780533667.816322 read 0x1986e4 thru 0x1a368e (0xafab bytes) 000402[1] 1780533667.840404 mapwrite 0x820191 thru 0x82fe4f (0xfcbf bytes) 000404[0] 1780533667.924529 mapread 0x305c57 thru 0x306df4 (0x119e bytes) 000406[0] 1780533667.967547 write 0x53143a thru 0x53f092 (0xdc59 bytes) 000408[1] 1780533667.991120 read 0x50144d thru 0x50d6d3 (0xc287 bytes) 000409[0] 1780533668.006106 read 0x367a53 thru 0x373df2 (0xc3a0 bytes) 000410[1] 1780533668.010305 mapwrite 0x529cce thru 0x532804 (0x8b37 bytes) 000411[0] 1780533668.067606 read 0x13b70f thru 0x1418d5 (0x61c7 bytes) 000412[1] 1780533668.075906 mapwrite 0x69a26d thru 0x6a345d (0x91f1 bytes) 000413[0] 1780533668.133089 write 0x56b680 thru 0x56e9ea (0x336b bytes) 000414[0] 1780533668.157967 mapwrite 0x88bc45 thru 0x89654b (0xa907 bytes) 000415[1] 1780533668.215299 write 0x3b8bc7 thru 0x3bffe1 (0x741b bytes) 000416[0] 1780533668.239252 write 0x02a133 thru 0x03149b (0x7369 bytes) 000417[0] 1780533668.263985 mapwrite 0x8c82e2 thru 0x8d2295 (0x9fb4 bytes) 000418[1] 1780533668.317769 trunc from 0x8d2296 to 0x283b75 (0x64e720 bytes) 000422[0] 1780533668.376643 mapwrite 0x5f6d26 thru 0x5f74ec (0x7c7 bytes) 000424[0] 1780533668.460752 read 0x0c06ee thru 0x0c37b2 (0x30c5 bytes) 000428[0] 1780533668.484194 read 0x21aa97 thru 0x2246b6 (0x9c20 bytes) 000432[0] 1780533668.491138 mapread 0x415ca0 thru 0x41ed71 (0x90d2 bytes) 000433[1] 1780533668.501360 read 0x0b2c4a thru 0x0b52c9 (0x2680 bytes) 000435[0] 1780533668.518260 mapwrite 0x7aba63 thru 0x7af24e (0x37ec bytes) 000436[1] 1780533668.572160 read 0x08e08c thru 0x090023 (0x1f98 bytes) 000437[0] 1780533668.591990 write 0x621b24 thru 0x630e84 (0xf361 bytes) 000438[1] 1780533668.607399 mapwrite 0x039fe9 thru 0x0458d4 (0xb8ec bytes) 000439[0] 1780533668.698275 read 0x73304b thru 0x733e80 (0xe36 bytes) 000440[1] 1780533668.716560 mapwrite 0x858ad1 thru 0x8632c3 (0xa7f3 bytes) 000444[1] 1780533668.819112 mapread 0x61db8f thru 0x627fc2 (0xa434 bytes) 000445[1] 1780533668.832713 mapread 0x09c874 thru 0x0abf29 (0xf6b6 bytes) 000446[1] 1780533668.862184 mapread 0x2cce46 thru 0x2d1a1e (0x4bd9 bytes) 000447[1] 1780533668.873599 read 0x7ffdcc thru 0x809541 (0x9776 bytes) 000448[0] 1780533668.878497 mapread 0x214657 thru 0x215b2d (0x14d7 bytes) 000449[0] 1780533668.893407 trunc from 0x8632c4 to 0x83bee9 (0x273da bytes) truncating to largest ever: 0x955c1c 000451[0] 1780533668.939538 trunc from 0x83beea to 0x955c1b (0x119d32 bytes) 000453[0] 1780533668.963110 write 0x8b4f74 thru 0x8bc0c8 (0x7155 bytes) 000455[1] 1780533668.981874 mapread 0x5ce34b thru 0x5d288c (0x4542 bytes) 000457[1] 1780533669.012293 mapwrite 0x9fac30 thru 0x9ff834 (0x4c05 bytes) 000458[1] 1780533669.093821 read 0x2d8a30 thru 0x2e4318 (0xb8e9 bytes) 000460[1] 1780533669.103563 mapread 0x0bcdf1 thru 0x0bee46 (0x2056 bytes) 000461[1] 1780533669.117530 read 0x3c5df5 thru 0x3cae80 (0x508c bytes) 000462[0] 1780533669.136288 write 0x15a315 thru 0x161a32 (0x771e bytes) 000463[0] 1780533669.160275 read 0x491a34 thru 0x495ec8 (0x4495 bytes) 000465[0] 1780533669.184430 write 0x1a9d31 thru 0x1aefc9 (0x5299 bytes) 000466[0] 1780533669.204496 mapwrite 0x2e53fa thru 0x2ef0a1 (0x9ca8 bytes) 000470[1] 1780533669.270296 mapwrite 0x655e1e thru 0x661dfc (0xbfdf bytes) 000471[1] 1780533669.334660 write 0x86fff0 thru 0x8793fc (0x940d bytes) 000472[0] 1780533669.354640 trunc from 0x9ff835 to 0x582710 (0x47d124 bytes) 000473[1] 1780533669.381784 write 0x8e8f37 thru 0x8f03f9 (0x74c3 bytes) 000474[0] 1780533669.401519 read 0x0a957a thru 0x0b450f (0xaf96 bytes) 000476[1] 1780533669.427508 read 0x02adc3 thru 0x0367fd (0xba3b bytes) 000479[0] 1780533669.444138 write 0x7c5f65 thru 0x7cc4ec (0x6588 bytes) 000481[1] 1780533669.458870 trunc from 0x8f03fa to 0x845f37 (0xaa4c2 bytes) 000483[0] 1780533669.498194 write 0x0620c6 thru 0x0677bf (0x56fa bytes) 000488[1] 1780533669.556213 mapwrite 0x2c7115 thru 0x2d3a74 (0xc960 bytes) 000492[0] 1780533669.653046 read 0x5097ca thru 0x50fa39 (0x6270 bytes) 000495[0] 1780533669.675482 mapread 0x5351be thru 0x53ca72 (0x78b5 bytes) 000496[0] 1780533669.698331 write 0x1029dd thru 0x103c53 (0x1277 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 3669 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[1] 1780533670.119974 mapwrite 0x615000 thru 0x615a80 (0xa81 bytes) truncating to largest ever: 0x485a3e 000004[0] 1780533670.174890 trunc from 0x615a81 to 0x485a3d (0x190043 bytes) truncating to largest ever: 0x593436 000005[0] 1780533670.199820 trunc from 0x485a3e to 0x593435 (0x10d9f8 bytes) 000007[1] 1780533670.230301 mapwrite 0x6ad000 thru 0x6b0065 (0x3066 bytes) 000009[1] 1780533670.313405 write 0x755000 thru 0x75fbf2 (0xabf3 bytes) 000010[1] 1780533670.317917 write 0x1c3000 thru 0x1cd73c (0xa73d bytes) 000014[0] 1780533670.357892 write_OD 0x4ec000 thru 0x4f3fff (0x8000 bytes) 000017[0] 1780533670.405631 write_OD 0x765000 thru 0x76bfff (0x7000 bytes) 000018[0] 1780533670.441941 trunc from 0x76c000 to 0x40260e (0x3699f1 bytes) 000019[0] 1780533670.459138 mapread 0x103000 thru 0x110dbd (0xddbe bytes) 000020[1] 1780533670.489922 read 0x1f1000 thru 0x1f4e50 (0x3e51 bytes) 000023[1] 1780533670.508262 write 0x4c4000 thru 0x4ce9c5 (0xa9c6 bytes) 000024[1] 1780533670.514167 trunc from 0x4ce9c6 to 0x174464 (0x35a561 bytes) 000027[1] 1780533670.552268 mapread 0x0c3000 thru 0x0d0d30 (0xdd31 bytes) 000030[0] 1780533670.596342 trunc from 0x174465 to 0x5240a0 (0x3afc3c bytes) 000032[0] 1780533670.614897 read_OD 0x34c000 thru 0x354fff (0x9000 bytes) truncating to largest ever: 0x65174e 000034[1] 1780533670.624979 trunc from 0x5240a1 to 0x65174d (0x12d6ad bytes) 000037[0] 1780533670.658697 write_OD 0x3bf000 thru 0x3ccfff (0xe000 bytes) 000038[1] 1780533670.705852 write 0x64e000 thru 0x658846 (0xa847 bytes) 000039[1] 1780533670.721426 trunc from 0x658847 to 0x0fe7e7 (0x55a05f bytes) 000050[0] 1780533670.822213 read_OD 0x090000 thru 0x097fff (0x8000 bytes) 000051[0] 1780533670.842600 mapread 0x0cb000 thru 0x0da0f8 (0xf0f9 bytes) 000052[0] 1780533670.863032 mapread 0x058000 thru 0x0625e8 (0xa5e9 bytes) 000055[1] 1780533670.864499 trunc from 0x0fe7e8 to 0x04099c (0xbde4b bytes) 000056[1] 1780533670.884571 trunc from 0x04099d to 0x1f92bd (0x1b8921 bytes) 000059[1] 1780533670.900975 mapwrite 0x706000 thru 0x70ac3a (0x4c3b bytes) 000062[0] 1780533670.974767 trunc from 0x70ac3b to 0x174344 (0x5968f6 bytes) 000064[0] 1780533671.003133 trunc from 0x174345 to 0x1ad4ba (0x39176 bytes) 000067[0] 1780533671.023622 read_OD 0x0a7000 thru 0x0b5fff (0xf000 bytes) truncating to largest ever: 0x68cb3c 000069[0] 1780533671.035216 trunc from 0x1ad4bb to 0x68cb3b (0x4df681 bytes) 000073[0] 1780533671.088347 mapwrite 0x2b7000 thru 0x2bec09 (0x7c0a bytes) 000076[0] 1780533671.191042 read_OD 0x330000 thru 0x337fff (0x8000 bytes) 000077[1] 1780533671.212248 read 0x023000 thru 0x02740d (0x440e bytes) 000085[0] 1780533671.225065 trunc from 0x68cb3c to 0x0cb2ac (0x5c188f bytes) 000088[1] 1780533671.244173 write 0x227000 thru 0x22e69d (0x769e bytes) 000092[0] 1780533671.284616 trunc from 0x22e69e to 0x03cfa1 (0x1f16fc bytes) truncating to largest ever: 0x8d9644 000096[0] 1780533671.312690 trunc from 0x03cfa2 to 0x8d9643 (0x89c6a2 bytes) 000097[1] 1780533671.325581 mapread 0x266000 thru 0x26b360 (0x5361 bytes) 000100[1] 1780533671.349252 mapwrite 0x2b9000 thru 0x2c3c5d (0xac5e bytes) 000102[0] 1780533671.416857 mapwrite 0x1b6000 thru 0x1b715f (0x1160 bytes) 000103[0] 1780533671.452374 read_OD 0x4bc000 thru 0x4c7fff (0xc000 bytes) 000104[0] 1780533671.462163 trunc from 0x8d9644 to 0x3c5fce (0x513675 bytes) 000106[1] 1780533671.477555 trunc from 0x3c5fcf to 0x0eae14 (0x2db1ba bytes) 000108[1] 1780533671.507193 mapwrite 0x7a2000 thru 0x7ae4f8 (0xc4f9 bytes) 000109[0] 1780533671.577814 write_OD 0x627000 thru 0x62bfff (0x5000 bytes) 000110[0] 1780533671.603119 read_OD 0x18c000 thru 0x190fff (0x5000 bytes) 000112[1] 1780533671.612338 mapwrite 0x7d8000 thru 0x7e62b3 (0xe2b4 bytes) 000113[0] 1780533671.690029 trunc from 0x7e62b4 to 0x7d98e5 (0xc9ce bytes) 000114[0] 1780533671.709697 mapwrite 0x00e000 thru 0x0119ae (0x39af bytes) 000118[1] 1780533671.787842 trunc from 0x7d98e6 to 0x19e59a (0x63b34b bytes) 000119[1] 1780533671.806225 read 0x022000 thru 0x02deee (0xbeef bytes) 000122[1] 1780533671.853581 write 0x26b000 thru 0x26bf4d (0xf4e bytes) 000124[0] 1780533671.864093 mapread 0x046000 thru 0x04b1cf (0x51d0 bytes) 000125[0] 1780533671.886144 mapwrite 0x430000 thru 0x432ff1 (0x2ff2 bytes) 000126[0] 1780533671.961071 write_OD 0x47e000 thru 0x480fff (0x3000 bytes) 000127[1] 1780533671.991365 write 0x8d7000 thru 0x8da99a (0x399b bytes) 000129[1] 1780533672.007621 trunc from 0x8da99b to 0x4c8e52 (0x411b48 bytes) 000133[1] 1780533672.029873 read 0x011000 thru 0x01aa9a (0x9a9b bytes) 000135[0] 1780533672.045668 mapread 0x24d000 thru 0x25abfa (0xdbfb bytes) 000138[1] 1780533672.073871 trunc from 0x4c8e53 to 0x371511 (0x157941 bytes) 000141[1] 1780533672.110307 read 0x34a000 thru 0x34ebbe (0x4bbf bytes) 000142[0] 1780533672.150225 read_OD 0x1d7000 thru 0x1dffff (0x9000 bytes) 000145[1] 1780533672.169501 trunc from 0x371512 to 0x45a02b (0xe8b1a bytes) 000146[0] 1780533672.205204 mapread 0x0a4000 thru 0x0abfbd (0x7fbe bytes) 000147[1] 1780533672.251271 mapwrite 0x2c1000 thru 0x2c400b (0x300c bytes) 000149[0] 1780533672.333555 write_OD 0x887000 thru 0x887fff (0x1000 bytes) 000150[1] 1780533672.366513 mapread 0x106000 thru 0x10ce75 (0x6e76 bytes) 000151[0] 1780533672.389362 mapwrite 0x3d5000 thru 0x3da687 (0x5688 bytes) 000152[1] 1780533672.434559 mapwrite 0x320000 thru 0x324881 (0x4882 bytes) 000154[0] 1780533672.505599 mapread 0x0b8000 thru 0x0c5046 (0xd047 bytes) 000156[0] 1780533672.573219 write_OD 0x6d6000 thru 0x6dbfff (0x6000 bytes) 000157[0] 1780533672.600283 trunc from 0x888000 to 0x692383 (0x1f5c7c bytes) 000158[0] 1780533672.618574 mapread 0x0a9000 thru 0x0b846c (0xf46d bytes) 000159[1] 1780533672.690392 mapwrite 0x6a2000 thru 0x6ae95d (0xc95e bytes) 000162[0] 1780533672.789364 trunc from 0x6ae95e to 0x597983 (0x116fda bytes) 000164[0] 1780533672.809453 read_OD 0x4f1000 thru 0x4f3fff (0x3000 bytes) 000169[0] 1780533672.827305 mapwrite 0x210000 thru 0x21671d (0x671e bytes) 000172[1] 1780533672.900019 mapread 0x00f000 thru 0x012cc6 (0x3cc7 bytes) 000173[1] 1780533672.931538 trunc from 0x597984 to 0x06be29 (0x52bb5a bytes) 000177[0] 1780533672.956685 read_OD 0x04a000 thru 0x051fff (0x8000 bytes) 000178[0] 1780533672.968962 write_OD 0x7d6000 thru 0x7dbfff (0x6000 bytes) 000182[0] 1780533673.006068 mapwrite 0x128000 thru 0x135338 (0xd339 bytes) 000183[1] 1780533673.094851 mapwrite 0x0af000 thru 0x0b503e (0x603f bytes) 000188[1] 1780533673.199368 write 0x69a000 thru 0x6a00ab (0x60ac bytes) 000189[1] 1780533673.211552 write 0x35e000 thru 0x364b08 (0x6b09 bytes) 000192[0] 1780533673.223483 read_OD 0x3ab000 thru 0x3b3fff (0x9000 bytes) 000193[0] 1780533673.241716 mapread 0x728000 thru 0x72ed0e (0x6d0f bytes) 000194[0] 1780533673.254382 write_OD 0x98a000 thru 0x98afff (0x1000 bytes) 000196[1] 1780533673.285764 write 0x927000 thru 0x931aa5 (0xaaa6 bytes) 000197[0] 1780533673.318172 read_OD 0x32c000 thru 0x333fff (0x8000 bytes) 000198[1] 1780533673.342055 mapread 0x71d000 thru 0x727514 (0xa515 bytes) 000201[0] 1780533673.379239 read_OD 0x092000 thru 0x09dfff (0xc000 bytes) 000202[1] 1780533673.384992 write 0x8e0000 thru 0x8ec659 (0xc65a bytes) 000203[1] 1780533673.395137 trunc from 0x98b000 to 0x7a27f5 (0x1e880a bytes) 000207[1] 1780533673.440254 write 0x4d5000 thru 0x4d7792 (0x2793 bytes) 000208[0] 1780533673.462156 mapread 0x42f000 thru 0x431e8c (0x2e8d bytes) 000211[0] 1780533673.492351 trunc from 0x7a27f6 to 0x3d8959 (0x3c9e9c bytes) 000215[0] 1780533673.529472 mapwrite 0x58a000 thru 0x595ef2 (0xbef3 bytes) 000217[1] 1780533673.612598 write 0x7ed000 thru 0x7f7a3c (0xaa3d bytes) 000218[0] 1780533673.632167 write_OD 0x85d000 thru 0x86bfff (0xf000 bytes) 000219[0] 1780533673.674196 read_OD 0x1e7000 thru 0x1edfff (0x7000 bytes) 000223[1] 1780533673.687319 read 0x109000 thru 0x116737 (0xd738 bytes) 000225[0] 1780533673.700293 mapwrite 0x500000 thru 0x501766 (0x1767 bytes) 000226[1] 1780533673.756992 read 0x30d000 thru 0x315663 (0x8664 bytes) 000227[1] 1780533673.773522 mapwrite 0x384000 thru 0x386a6d (0x2a6e bytes) 000228[0] 1780533673.868536 read_OD 0x00d000 thru 0x017fff (0xb000 bytes) 000232[1] 1780533673.892657 trunc from 0x86c000 to 0x3812bf (0x4ead40 bytes) 000233[0] 1780533673.909073 mapread 0x224000 thru 0x22a5e2 (0x65e3 bytes) 000242[1] 1780533673.955914 write 0x0b2000 thru 0x0b88db (0x68dc bytes) 000243[1] 1780533673.964924 read 0x0ad000 thru 0x0b6569 (0x956a bytes) 000244[0] 1780533674.003178 trunc from 0x3812c0 to 0x127c7e (0x259641 bytes) 000245[1] 1780533674.036703 read 0x0c8000 thru 0x0d7c97 (0xfc98 bytes) 000246[0] 1780533674.063716 mapread 0x0a7000 thru 0x0b1d2d (0xad2e bytes) 000247[0] 1780533674.079033 mapwrite 0x536000 thru 0x5439d0 (0xd9d1 bytes) 000249[0] 1780533674.155693 write_OD 0x4c1000 thru 0x4cdfff (0xd000 bytes) 000250[0] 1780533674.189012 read_OD 0x2db000 thru 0x2e0fff (0x6000 bytes) 000254[0] 1780533674.200912 mapread 0x20c000 thru 0x219976 (0xd977 bytes) truncating to largest ever: 0x95b3b1 000255[0] 1780533674.218224 trunc from 0x5439d1 to 0x95b3b0 (0x4179e0 bytes) 000256[1] 1780533674.245422 mapread 0x00a000 thru 0x00cec5 (0x2ec6 bytes) 000257[0] 1780533674.289359 read_OD 0x802000 thru 0x80ffff (0xe000 bytes) 000261[0] 1780533674.294633 read_OD 0x25d000 thru 0x266fff (0xa000 bytes) 000263[1] 1780533674.299346 mapread 0x8f5000 thru 0x8fedfb (0x9dfc bytes) 000264[0] 1780533674.345065 read_OD 0x62b000 thru 0x635fff (0xb000 bytes) 000266[1] 1780533674.350640 trunc from 0x95b3b1 to 0x792c7c (0x1c8734 bytes) truncating to largest ever: 0x9b2689 000268[1] 1780533674.393657 trunc from 0x792c7d to 0x9b2688 (0x21fa0c bytes) 000269[0] 1780533674.410582 trunc from 0x9b2689 to 0x3b119e (0x6014ea bytes) 000276[1] 1780533674.500856 read 0x001000 thru 0x0015e1 (0x5e2 bytes) 000280[0] 1780533674.551585 write_OD 0x179000 thru 0x185fff (0xd000 bytes) 000282[1] 1780533674.587595 trunc from 0x3b119f to 0x7d33c0 (0x422222 bytes) 000283[1] 1780533674.610266 mapwrite 0x4a0000 thru 0x4adb22 (0xdb23 bytes) 000288[1] 1780533674.689290 write 0x6e9000 thru 0x6ea5cb (0x15cc bytes) 000289[0] 1780533674.698603 write_OD 0x25b000 thru 0x263fff (0x9000 bytes) 000291[0] 1780533674.740099 trunc from 0x7d33c1 to 0x0485a4 (0x78ae1c bytes) 000292[0] 1780533674.772061 trunc from 0x0485a5 to 0x3368f8 (0x2ee354 bytes) 000294[1] 1780533674.797547 write 0x059000 thru 0x0624c4 (0x94c5 bytes) 000295[1] 1780533674.807836 write 0x704000 thru 0x70f7f7 (0xb7f8 bytes) 000297[1] 1780533674.831049 trunc from 0x70f7f8 to 0x8869b4 (0x1771bd bytes) 000299[1] 1780533674.855338 read 0x421000 thru 0x42c2e1 (0xb2e2 bytes) 000305[0] 1780533674.885110 read_OD 0x2c4000 thru 0x2d0fff (0xd000 bytes) 000306[1] 1780533674.899359 read 0x340000 thru 0x341c49 (0x1c4a bytes) 000309[1] 1780533674.904766 read 0x49e000 thru 0x4a7f49 (0x9f4a bytes) 000310[1] 1780533674.909769 mapread 0x755000 thru 0x756b14 (0x1b15 bytes) 000312[0] 1780533674.916130 write_OD 0x850000 thru 0x85cfff (0xd000 bytes) 000316[1] 1780533674.950381 mapwrite 0x6f0000 thru 0x6f6ac4 (0x6ac5 bytes) 000318[1] 1780533675.005654 mapread 0x862000 thru 0x8675e0 (0x55e1 bytes) 000320[0] 1780533675.027761 write_OD 0x620000 thru 0x62efff (0xf000 bytes) 000322[1] 1780533675.054673 write 0x4b5000 thru 0x4c0f1b (0xbf1c bytes) 000324[0] 1780533675.075494 mapwrite 0x7ac000 thru 0x7adab8 (0x1ab9 bytes) 000326[0] 1780533675.133685 trunc from 0x8869b5 to 0x9059a5 (0x7eff1 bytes) 000327[0] 1780533675.142431 write_OD 0x121000 thru 0x12efff (0xe000 bytes) 000329[0] 1780533675.171087 write_OD 0x877000 thru 0x881fff (0xb000 bytes) 000332[0] 1780533675.197230 write_OD 0x1d3000 thru 0x1ddfff (0xb000 bytes) 000335[0] 1780533675.224224 trunc from 0x9059a6 to 0x9212f2 (0x1b94d bytes) 000336[1] 1780533675.238584 trunc from 0x9212f3 to 0x4d0662 (0x450c90 bytes) 000337[1] 1780533675.252238 read 0x388000 thru 0x390844 (0x8845 bytes) 000338[1] 1780533675.257334 trunc from 0x4d0663 to 0x6b849f (0x1e7e3d bytes) 000344[0] 1780533675.300766 mapread 0x657000 thru 0x65e547 (0x7548 bytes) 000345[1] 1780533675.317076 read 0x693000 thru 0x694eab (0x1eac bytes) 000346[0] 1780533675.321533 trunc from 0x6b84a0 to 0x6f9d98 (0x418f9 bytes) 000347[1] 1780533675.344909 trunc from 0x6f9d99 to 0x3ce713 (0x32b685 bytes) 000349[0] 1780533675.377222 trunc from 0x3ce714 to 0x72868c (0x359f79 bytes) 000351[0] 1780533675.406544 read_OD 0x44e000 thru 0x457fff (0xa000 bytes) 000352[1] 1780533675.420916 trunc from 0x72868d to 0x959a2b (0x23139f bytes) 000353[0] 1780533675.443776 trunc from 0x959a2c to 0x6cc3e2 (0x28d649 bytes) 000355[1] 1780533675.462576 read 0x57a000 thru 0x57fa9f (0x5aa0 bytes) 000356[0] 1780533675.469557 write_OD 0x3fa000 thru 0x3fafff (0x1000 bytes) 000357[1] 1780533675.492287 write 0x8cc000 thru 0x8cfd3d (0x3d3e bytes) 000361[1] 1780533675.514130 mapread 0x37c000 thru 0x387ac8 (0xbac9 bytes) 000362[1] 1780533675.535614 trunc from 0x8cfd3e to 0x2da65e (0x5f56df bytes) 000363[0] 1780533675.549295 mapread 0x205000 thru 0x20ee65 (0x9e66 bytes) 000365[1] 1780533675.567963 mapwrite 0x60e000 thru 0x61ddea (0xfdeb bytes) 000367[1] 1780533675.633600 read 0x2b8000 thru 0x2bab5d (0x2b5e bytes) 000369[0] 1780533675.645753 mapread 0x00f000 thru 0x01338c (0x438d bytes) 000371[0] 1780533675.659458 mapread 0x19b000 thru 0x1a474e (0x974f bytes) 000372[1] 1780533675.665890 write 0x0d3000 thru 0x0d8d73 (0x5d74 bytes) 000373[0] 1780533675.677277 write_OD 0x20c000 thru 0x20ffff (0x4000 bytes) 000376[1] 1780533675.700406 mapwrite 0x1d3000 thru 0x1d4892 (0x1893 bytes) 000378[0] 1780533675.734276 mapread 0x430000 thru 0x437321 (0x7322 bytes) 000379[0] 1780533675.750374 mapread 0x287000 thru 0x292bec (0xbbed bytes) 000380[1] 1780533675.761905 mapread 0x576000 thru 0x57e320 (0x8321 bytes) 000384[1] 1780533675.778639 trunc from 0x61ddeb to 0x4fb244 (0x122ba6 bytes) 000385[1] 1780533675.800892 mapwrite 0x1b6000 thru 0x1c10f1 (0xb0f2 bytes) 000386[1] 1780533675.838936 mapread 0x16c000 thru 0x171d48 (0x5d49 bytes) 000389[1] 1780533675.857399 mapwrite 0x304000 thru 0x308f29 (0x4f2a bytes) 000390[1] 1780533675.885203 mapread 0x300000 thru 0x30dcd4 (0xdcd5 bytes) 000391[0] 1780533675.892279 read_OD 0x035000 thru 0x03afff (0x6000 bytes) 000392[1] 1780533675.903001 trunc from 0x4fb245 to 0x4070cc (0xf4178 bytes) 000395[0] 1780533675.923449 write_OD 0x7a4000 thru 0x7a8fff (0x5000 bytes) 000398[1] 1780533675.946723 mapread 0x6b0000 thru 0x6b4a10 (0x4a11 bytes) 000399[0] 1780533675.954438 mapwrite 0x3d0000 thru 0x3de38e (0xe38f bytes) 000401[0] 1780533675.999111 trunc from 0x7a9000 to 0x08c28f (0x71cd70 bytes) 000402[1] 1780533676.013840 mapread 0x017000 thru 0x01b438 (0x4439 bytes) 000403[0] 1780533676.034800 mapread 0x075000 thru 0x0773ca (0x23cb bytes) 000404[0] 1780533676.050172 trunc from 0x08c290 to 0x21a2a7 (0x18e018 bytes) 000405[0] 1780533676.068367 mapwrite 0x210000 thru 0x210fb1 (0xfb2 bytes) 000406[1] 1780533676.094244 trunc from 0x21a2a8 to 0x18b77d (0x8eb2a bytes) 000408[1] 1780533676.113203 read 0x14c000 thru 0x14c76a (0x76b bytes) 000409[0] 1780533676.126688 write_OD 0x652000 thru 0x654fff (0x3000 bytes) 000414[0] 1780533676.149780 write_OD 0x5ca000 thru 0x5d8fff (0xf000 bytes) 000415[1] 1780533676.171889 read 0x0c5000 thru 0x0cb862 (0x6863 bytes) 000416[0] 1780533676.176155 read_OD 0x2c9000 thru 0x2d4fff (0xc000 bytes) 000417[1] 1780533676.183096 write 0x540000 thru 0x545320 (0x5321 bytes) 000418[1] 1780533676.190707 mapwrite 0x3f3000 thru 0x3f9cf0 (0x6cf1 bytes) 000419[1] 1780533676.225089 mapread 0x574000 thru 0x5826f3 (0xe6f4 bytes) 000421[0] 1780533676.245124 write_OD 0x722000 thru 0x72ffff (0xe000 bytes) 000423[1] 1780533676.267950 write 0x48b000 thru 0x495e05 (0xae06 bytes) 000424[0] 1780533676.283508 read_OD 0x52b000 thru 0x538fff (0xe000 bytes) 000428[0] 1780533676.302290 read_OD 0x5ab000 thru 0x5affff (0x5000 bytes) 000432[0] 1780533676.307548 mapread 0x3d4000 thru 0x3d8982 (0x4983 bytes) 000434[1] 1780533676.318919 trunc from 0x730000 to 0x8596bd (0x1296be bytes) 000436[0] 1780533676.348182 read_OD 0x52c000 thru 0x531fff (0x6000 bytes) 000439[0] 1780533676.363419 read_OD 0x803000 thru 0x803fff (0x1000 bytes) 000447[1] 1780533676.368551 write 0x6dc000 thru 0x6dc7c3 (0x7c4 bytes) 000448[0] 1780533676.382950 read_OD 0x6f5000 thru 0x6f5fff (0x1000 bytes) 000449[1] 1780533676.398844 mapread 0x562000 thru 0x56ed16 (0xcd17 bytes) 000450[0] 1780533676.413441 write_OD 0x41c000 thru 0x423fff (0x8000 bytes) 000451[1] 1780533676.436906 trunc from 0x8596be to 0x196c12 (0x6c2aab bytes) 000453[0] 1780533676.457915 mapwrite 0x095000 thru 0x09fb8f (0xab90 bytes) 000454[0] 1780533676.504116 write_OD 0x5e2000 thru 0x5eefff (0xd000 bytes) 000455[0] 1780533676.534129 mapwrite 0x9f6000 thru 0x9ff94f (0x9950 bytes) 000458[0] 1780533676.588470 trunc from 0x9ff950 to 0x7bc57a (0x2433d5 bytes) 000460[1] 1780533676.611316 trunc from 0x7bc57b to 0x154a7a (0x667b00 bytes) 000461[0] 1780533676.628528 mapread 0x0d1000 thru 0x0db56e (0xa56f bytes) 000466[0] 1780533676.653701 trunc from 0x154a7b to 0x40abe5 (0x2b616b bytes) 000467[1] 1780533676.667795 trunc from 0x40abe6 to 0x2b3a75 (0x157170 bytes) 000469[1] 1780533676.691256 write 0x477000 thru 0x480552 (0x9553 bytes) 000470[1] 1780533676.693293 trunc from 0x480553 to 0x5c730a (0x146db8 bytes) 000471[0] 1780533676.709353 read_OD 0x391000 thru 0x391fff (0x1000 bytes) 000472[0] 1780533676.723457 trunc from 0x5c730b to 0x7fdb37 (0x23682d bytes) 000473[0] 1780533676.741935 write_OD 0x90d000 thru 0x912fff (0x6000 bytes) 000474[0] 1780533676.772035 mapwrite 0x478000 thru 0x479bbe (0x1bbf bytes) 000475[0] 1780533676.823979 mapwrite 0x775000 thru 0x778938 (0x3939 bytes) 000476[1] 1780533676.871823 write 0x0db000 thru 0x0e0008 (0x5009 bytes) 000477[1] 1780533676.880644 mapread 0x832000 thru 0x83ea28 (0xca29 bytes) 000478[1] 1780533676.897495 mapwrite 0x8b8000 thru 0x8c498d (0xc98e bytes) 000479[0] 1780533676.941006 write_OD 0x7b7000 thru 0x7b8fff (0x2000 bytes) 000481[0] 1780533676.966806 write_OD 0x996000 thru 0x99ffff (0xa000 bytes) 000482[0] 1780533676.999008 read_OD 0x875000 thru 0x87afff (0x6000 bytes) 000485[1] 1780533677.006444 mapwrite 0x634000 thru 0x637490 (0x3491 bytes) 000486[1] 1780533677.049570 trunc from 0x9a0000 to 0x86357c (0x13ca83 bytes) 000492[1] 1780533677.098878 mapread 0x71f000 thru 0x72c6e6 (0xd6e7 bytes) 000494[0] 1780533677.120343 mapread 0x43b000 thru 0x43bcd2 (0xcd3 bytes) 000497[1] 1780533677.135080 read 0x2d2000 thru 0x2d2fb6 (0xfb7 bytes) 000498[1] 1780533677.142199 read 0x78d000 thru 0x7974f4 (0xa4f5 bytes) 000500[1] 1780533677.150541 read 0x04d000 thru 0x05b5c4 (0xe5c5 bytes) All operations completed A-OK! PASS 16a (22s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 20:41:19 (1780533679) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.129118 s, 65.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.116385 s, 72.1 MB/s Chance of close/open is 1 in 50 Seed set to 3680 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1780533680.438549 mapwrite 0x48bb46 thru 0x48bc99 (0x154 bytes) truncating to largest ever: 0x27fc81 000002[1] 1780533680.513252 trunc from 0x48bc9a to 0x27fc80 (0x20c019 bytes) 000003[0] 1780533680.530244 mapwrite 0x04d870 thru 0x04dc79 (0x40a bytes) 000008[1] 1780533680.619484 mapread 0x189ec7 thru 0x18a752 (0x88c bytes) 000014[0] 1780533680.651393 mapwrite 0x5a314c thru 0x5a4970 (0x1825 bytes) 000018[1] 1780533680.721627 read 0x5a477f thru 0x5a4970 (0x1f2 bytes) 000020[0] 1780533680.739445 mapread 0x27dd06 thru 0x27ef84 (0x127f bytes) 000022[0] 1780533680.755369 mapwrite 0x1f6133 thru 0x1f6da3 (0xc71 bytes) 000023[0] 1780533680.782555 mapread 0x5172ab thru 0x5182ea (0x1040 bytes) 000024[1] 1780533680.789136 read 0x3b47a6 thru 0x3b5516 (0xd71 bytes) 000026[1] 1780533680.799626 mapread 0x27dee2 thru 0x27f82d (0x194c bytes) 000028[0] 1780533680.810544 mapwrite 0x7b42e5 thru 0x7b5b0c (0x1828 bytes) 000029[1] 1780533680.870513 mapwrite 0x4fd529 thru 0x4ff423 (0x1efb bytes) truncating to largest ever: 0x918133 000034[0] 1780533680.961524 trunc from 0x7b5b0d to 0x918132 (0x162626 bytes) 000037[0] 1780533680.973910 write 0x2dd24d thru 0x2de558 (0x130c bytes) 000038[0] 1780533680.978531 mapwrite 0x0f3ccb thru 0x0f4f98 (0x12ce bytes) 000039[0] 1780533681.028180 mapread 0x8bce74 thru 0x8bd598 (0x725 bytes) 000040[1] 1780533681.035670 mapwrite 0x443145 thru 0x443dc5 (0xc81 bytes) 000041[1] 1780533681.080627 mapread 0x867f80 thru 0x868d73 (0xdf4 bytes) truncating to largest ever: 0x9aaebb 000044[1] 1780533681.105005 trunc from 0x918133 to 0x9aaeba (0x92d88 bytes) 000045[0] 1780533681.131573 read 0x650763 thru 0x650956 (0x1f4 bytes) 000046[0] 1780533681.148517 write 0x192cf6 thru 0x194972 (0x1c7d bytes) 000057[1] 1780533681.211429 read 0x8f1d0e thru 0x8f296d (0xc60 bytes) 000058[0] 1780533681.223827 read 0x3b13f3 thru 0x3b1e60 (0xa6e bytes) 000064[1] 1780533681.229106 write 0x1075b4 thru 0x107bf9 (0x646 bytes) 000069[1] 1780533681.264030 trunc from 0x9aaebb to 0x76bd81 (0x23f139 bytes) 000070[1] 1780533681.296156 read 0x174236 thru 0x174bea (0x9b5 bytes) 000071[0] 1780533681.301670 write 0x9f1981 thru 0x9f2667 (0xce7 bytes) 000072[0] 1780533681.329264 mapwrite 0x8c9f6d thru 0x8cbc5d (0x1cf1 bytes) 000076[1] 1780533681.411208 mapread 0x744a68 thru 0x74670c (0x1ca5 bytes) 000077[0] 1780533681.435152 trunc from 0x9f2668 to 0x22da39 (0x7c4c2e bytes) 000078[0] 1780533681.450481 read 0x2224a3 thru 0x22374f (0x12ad bytes) 000079[0] 1780533681.473003 read 0x09eaac thru 0x09ec8a (0x1df bytes) 000080[0] 1780533681.491469 write 0x0cc45d thru 0x0ccad7 (0x67b bytes) 000082[0] 1780533681.508668 read 0x1c6db4 thru 0x1c8768 (0x19b5 bytes) 000084[1] 1780533681.509665 read 0x19f8c7 thru 0x1a0959 (0x1093 bytes) 000086[0] 1780533681.566627 mapwrite 0x5fbfa7 thru 0x5fc4d8 (0x532 bytes) 000089[1] 1780533681.697002 read 0x5f2bc8 thru 0x5f4252 (0x168b bytes) 000090[0] 1780533681.708700 write 0x2ca176 thru 0x2ca9b8 (0x843 bytes) 000091[1] 1780533681.709631 mapwrite 0x495f81 thru 0x496ff1 (0x1071 bytes) 000092[0] 1780533681.747647 mapread 0x3c2f03 thru 0x3c330f (0x40d bytes) 000093[1] 1780533681.760646 mapwrite 0x0ab12f thru 0x0ac7b5 (0x1687 bytes) 000094[1] 1780533681.822514 read 0x059e65 thru 0x05a8c5 (0xa61 bytes) 000095[1] 1780533681.840357 read 0x38b143 thru 0x38c435 (0x12f3 bytes) 000098[1] 1780533681.865131 trunc from 0x5fc4d9 to 0x864315 (0x267e3d bytes) 000099[0] 1780533681.879060 mapwrite 0x206576 thru 0x2078ae (0x1339 bytes) 000103[0] 1780533681.966495 write 0x341457 thru 0x3429f4 (0x159e bytes) 000108[0] 1780533682.022886 read 0x6929c8 thru 0x6948b0 (0x1ee9 bytes) 000111[1] 1780533682.039691 mapwrite 0x17cef7 thru 0x17deb2 (0xfbc bytes) 000113[0] 1780533682.078317 trunc from 0x864316 to 0x675d05 (0x1ee610 bytes) 000114[1] 1780533682.097257 mapread 0x2b81bb thru 0x2b8dd8 (0xc1e bytes) 000117[0] 1780533682.120462 mapwrite 0x174282 thru 0x174b11 (0x890 bytes) 000119[0] 1780533682.145771 write 0x03fd8e thru 0x040942 (0xbb5 bytes) 000120[0] 1780533682.165901 mapwrite 0x393a37 thru 0x394171 (0x73b bytes) 000121[1] 1780533682.195590 read 0x467690 thru 0x468a9c (0x140d bytes) 000123[0] 1780533682.208417 write 0x45d066 thru 0x45e90b (0x18a6 bytes) 000124[0] 1780533682.230019 write 0x5b609c thru 0x5b7cd2 (0x1c37 bytes) 000125[0] 1780533682.235440 read 0x1878bf thru 0x188103 (0x845 bytes) 000126[1] 1780533682.239114 trunc from 0x675d06 to 0x33aa84 (0x33b281 bytes) 000127[0] 1780533682.269565 mapread 0x065fe4 thru 0x066e56 (0xe73 bytes) 000130[0] 1780533682.297395 trunc from 0x33aa85 to 0x973ea3 (0x63941f bytes) 000132[0] 1780533682.316434 mapwrite 0x2d80c3 thru 0x2d9fd3 (0x1f11 bytes) 000133[1] 1780533682.351992 read 0x07c9b1 thru 0x07dc5b (0x12ab bytes) 000134[1] 1780533682.362701 trunc from 0x973ea4 to 0x5d71ec (0x39ccb7 bytes) 000136[1] 1780533682.393888 mapread 0x37ca4a thru 0x37ce83 (0x43a bytes) 000137[1] 1780533682.405371 read 0x2f04c6 thru 0x2f1967 (0x14a2 bytes) 000140[1] 1780533682.412358 mapwrite 0x136b45 thru 0x137b3f (0xffb bytes) 000141[1] 1780533682.434449 read 0x0086ec thru 0x008c77 (0x58c bytes) 000144[0] 1780533682.454822 mapread 0x4e013c thru 0x4e122c (0x10f1 bytes) 000147[1] 1780533682.480027 write 0x6da2b2 thru 0x6db041 (0xd90 bytes) 000150[0] 1780533682.496510 mapread 0x6d2281 thru 0x6d288e (0x60e bytes) 000152[1] 1780533682.519876 write 0x79e784 thru 0x79fe86 (0x1703 bytes) 000153[1] 1780533682.532951 write 0x5f1afc thru 0x5f1d0d (0x212 bytes) 000154[1] 1780533682.544193 mapwrite 0x1aab4e thru 0x1ab5d2 (0xa85 bytes) 000157[0] 1780533682.595711 write 0x84bc23 thru 0x84da70 (0x1e4e bytes) 000161[1] 1780533682.634805 mapread 0x816637 thru 0x8167fc (0x1c6 bytes) 000162[1] 1780533682.654272 write 0x7f0e33 thru 0x7f284d (0x1a1b bytes) 000163[1] 1780533682.667658 write 0x27ec76 thru 0x27fedf (0x126a bytes) 000164[1] 1780533682.685267 trunc from 0x84da71 to 0x5d7e1a (0x275c56 bytes) 000165[1] 1780533682.703899 write 0x016f2c thru 0x017542 (0x617 bytes) 000166[0] 1780533682.711106 read 0x11767a thru 0x1192ec (0x1c73 bytes) 000168[1] 1780533682.721579 mapread 0x0bb5be thru 0x0bcdef (0x1832 bytes) 000170[0] 1780533682.731381 trunc from 0x5d7e1b to 0x0c4b24 (0x5132f6 bytes) 000174[0] 1780533682.746680 read 0x092d10 thru 0x093491 (0x782 bytes) 000175[1] 1780533682.763635 mapread 0x026eff thru 0x028a78 (0x1b7a bytes) 000176[1] 1780533682.784142 mapwrite 0x75569e thru 0x756fd5 (0x1938 bytes) 000178[1] 1780533682.839765 read 0x5e5143 thru 0x5e66d5 (0x1593 bytes) 000179[1] 1780533682.843938 mapwrite 0x283fd1 thru 0x2857bf (0x17ef bytes) 000180[1] 1780533682.864014 mapwrite 0x200673 thru 0x201577 (0xf05 bytes) 000181[0] 1780533682.886236 mapread 0x45bf88 thru 0x45c0ac (0x125 bytes) 000182[1] 1780533682.900892 write 0x6c6bae thru 0x6c8a25 (0x1e78 bytes) 000184[1] 1780533682.919547 write 0x76480a thru 0x765fee (0x17e5 bytes) 000185[1] 1780533682.930476 mapread 0x3988a2 thru 0x398929 (0x88 bytes) 000186[1] 1780533682.934614 write 0x3e6b38 thru 0x3e7222 (0x6eb bytes) 000187[0] 1780533682.941861 read 0x1ce89a thru 0x1cfb5e (0x12c5 bytes) 000189[1] 1780533682.974211 mapwrite 0x258d96 thru 0x259b08 (0xd73 bytes) 000190[1] 1780533683.013702 trunc from 0x765fef to 0x895d24 (0x12fd36 bytes) 000192[0] 1780533683.034651 trunc from 0x895d25 to 0x39e96f (0x4f73b5 bytes) 000194[0] 1780533683.067552 trunc from 0x39e970 to 0x79c5cc (0x3fdc5d bytes) 000196[1] 1780533683.089994 mapwrite 0x940e7c thru 0x94152a (0x6af bytes) 000204[1] 1780533683.158108 mapwrite 0x4a1a12 thru 0x4a350d (0x1afc bytes) 000205[1] 1780533683.198172 trunc from 0x94152b to 0x46fca4 (0x4d1886 bytes) 000206[0] 1780533683.208274 mapread 0x32fdd3 thru 0x330f62 (0x1190 bytes) 000207[1] 1780533683.225583 read 0x08257c thru 0x082a3c (0x4c1 bytes) 000208[1] 1780533683.228150 trunc from 0x46fca5 to 0x800ec1 (0x39121d bytes) 000214[0] 1780533683.261392 mapread 0x65d4b0 thru 0x65e9e7 (0x1538 bytes) 000215[1] 1780533683.287250 trunc from 0x800ec2 to 0x3d809c (0x428e25 bytes) 000216[1] 1780533683.300975 read 0x306bbe thru 0x3074da (0x91d bytes) 000217[0] 1780533683.336906 mapwrite 0x5a2727 thru 0x5a327d (0xb57 bytes) 000220[1] 1780533683.389597 trunc from 0x5a327e to 0x1e26e1 (0x3c0b9c bytes) 000221[1] 1780533683.410049 write 0x7da456 thru 0x7db46a (0x1015 bytes) 000222[1] 1780533683.421071 read 0x5adecd thru 0x5af75a (0x188e bytes) 000230[0] 1780533683.444227 trunc from 0x7db46b to 0x4cf840 (0x30bc2a bytes) 000231[1] 1780533683.459884 write 0x31abb8 thru 0x31c833 (0x1c7c bytes) 000232[1] 1780533683.471428 mapread 0x42f6cc thru 0x430032 (0x967 bytes) 000233[0] 1780533683.487569 write 0x8372a5 thru 0x838fd3 (0x1d2f bytes) 000234[0] 1780533683.500685 write 0x16cde1 thru 0x16e3ac (0x15cc bytes) 000236[0] 1780533683.514953 read 0x277932 thru 0x278613 (0xce2 bytes) 000242[0] 1780533683.529179 write 0x638ba7 thru 0x63a61c (0x1a76 bytes) 000245[0] 1780533683.545898 write 0x5ec998 thru 0x5ed57c (0xbe5 bytes) 000246[1] 1780533683.550463 mapread 0x6355fe thru 0x636097 (0xa9a bytes) 000247[0] 1780533683.573158 write 0x03feaa thru 0x041cc1 (0x1e18 bytes) 000254[0] 1780533683.603325 read 0x74808d thru 0x749bd2 (0x1b46 bytes) 000255[0] 1780533683.608890 read 0x6f962d thru 0x6faad2 (0x14a6 bytes) 000257[0] 1780533683.614773 read 0x4a56fb thru 0x4a7614 (0x1f1a bytes) 000258[1] 1780533683.625989 write 0x5c74ea thru 0x5c8f0b (0x1a22 bytes) 000259[1] 1780533683.643486 write 0x030313 thru 0x030fe0 (0xcce bytes) 000262[1] 1780533683.657464 mapwrite 0x57fc59 thru 0x580bc2 (0xf6a bytes) 000263[0] 1780533683.688722 read 0x0e9cd8 thru 0x0eb417 (0x1740 bytes) 000265[1] 1780533683.711684 read 0x3af070 thru 0x3af253 (0x1e4 bytes) 000269[1] 1780533683.733570 mapread 0x1e2f0a thru 0x1e3b61 (0xc58 bytes) 000270[0] 1780533683.737709 trunc from 0x838fd4 to 0x4076d1 (0x431902 bytes) 000272[1] 1780533683.760970 mapread 0x20af0f thru 0x20bacf (0xbc1 bytes) 000275[1] 1780533683.773012 mapread 0x100dd7 thru 0x102afc (0x1d26 bytes) 000281[1] 1780533683.806879 write 0x1df329 thru 0x1dfffa (0xcd2 bytes) 000282[1] 1780533683.820029 write 0x2cc3ec thru 0x2cd95a (0x156f bytes) 000283[1] 1780533683.828438 mapwrite 0x2b378d thru 0x2b540e (0x1c82 bytes) 000288[0] 1780533683.897913 read 0x2d4cdf thru 0x2d6081 (0x13a3 bytes) 000290[0] 1780533683.906175 mapread 0x02bef4 thru 0x02dc04 (0x1d11 bytes) 000291[0] 1780533683.926361 trunc from 0x4076d2 to 0x9aae34 (0x5a3763 bytes) 000292[0] 1780533683.951391 mapread 0x94fb24 thru 0x951aaf (0x1f8c bytes) 000295[0] 1780533683.977843 mapread 0x0eee7b thru 0x0ef81b (0x9a1 bytes) 000297[0] 1780533684.003222 read 0x8ebb48 thru 0x8ec010 (0x4c9 bytes) 000300[1] 1780533684.039805 write 0x393122 thru 0x39490e (0x17ed bytes) 000301[0] 1780533684.068202 write 0x8c04aa thru 0x8c2301 (0x1e58 bytes) 000302[0] 1780533684.092195 trunc from 0x9aae35 to 0x0d2de8 (0x8d804c bytes) 000305[0] 1780533684.113367 mapwrite 0x19c68b thru 0x19d1a7 (0xb1d bytes) 000307[0] 1780533684.163203 trunc from 0x19d1a8 to 0x45f03e (0x2c1e97 bytes) 000309[1] 1780533684.173906 write 0x841e67 thru 0x842c33 (0xdcd bytes) 000312[0] 1780533684.201668 write 0x8596e1 thru 0x859da2 (0x6c2 bytes) 000313[0] 1780533684.223942 write 0x9cc7d7 thru 0x9cde3a (0x1664 bytes) 000314[1] 1780533684.233246 read 0x51e82b thru 0x51eb5d (0x333 bytes) 000317[0] 1780533684.252353 write 0x3aa06b thru 0x3ab458 (0x13ee bytes) 000318[1] 1780533684.263241 mapwrite 0x86e216 thru 0x86e3bf (0x1aa bytes) 000319[0] 1780533684.310982 write 0x9d0491 thru 0x9d15da (0x114a bytes) 000320[1] 1780533684.315216 write 0x462b8a thru 0x464a7c (0x1ef3 bytes) 000321[1] 1780533684.328177 read 0x5c95ae thru 0x5c9835 (0x288 bytes) 000323[1] 1780533684.346401 mapread 0x937ec2 thru 0x93928d (0x13cc bytes) 000325[1] 1780533684.374030 write 0x8335d1 thru 0x833cc5 (0x6f5 bytes) 000326[1] 1780533684.383474 mapwrite 0x0e38f7 thru 0x0e4393 (0xa9d bytes) 000328[0] 1780533684.419890 mapwrite 0x550f0a thru 0x552428 (0x151f bytes) 000330[0] 1780533684.496982 mapwrite 0x1c09fb thru 0x1c1985 (0xf8b bytes) 000331[0] 1780533684.531990 mapread 0x0e7ec0 thru 0x0e89e5 (0xb26 bytes) 000332[1] 1780533684.537975 trunc from 0x9d15db to 0x974342 (0x5d298 bytes) 000333[1] 1780533684.554185 mapread 0x9515a5 thru 0x951b79 (0x5d5 bytes) 000336[1] 1780533684.596251 read 0x6dfb1b thru 0x6e0b93 (0x1079 bytes) 000337[1] 1780533684.600629 trunc from 0x974343 to 0x1b2349 (0x7c1ff9 bytes) 000339[0] 1780533684.635140 mapwrite 0x9d093e thru 0x9d2525 (0x1be8 bytes) 000340[0] 1780533684.700361 write 0x415926 thru 0x415f58 (0x633 bytes) 000342[1] 1780533684.709901 mapwrite 0x14c7b1 thru 0x14e76d (0x1fbd bytes) 000344[0] 1780533684.775037 read 0x2bb9ac thru 0x2bbb68 (0x1bd bytes) 000346[0] 1780533684.794160 write 0x06581e thru 0x067553 (0x1d36 bytes) 000354[0] 1780533684.833138 mapwrite 0x298cb7 thru 0x29900a (0x354 bytes) 000356[0] 1780533684.859423 read 0x1452a0 thru 0x146d6b (0x1acc bytes) 000359[0] 1780533684.872913 mapread 0x2f1cb4 thru 0x2f24bc (0x809 bytes) 000361[0] 1780533684.888592 mapwrite 0x071bc1 thru 0x072290 (0x6d0 bytes) 000364[1] 1780533684.927591 mapwrite 0x7b569b thru 0x7b6ed2 (0x1838 bytes) 000367[0] 1780533684.990932 mapread 0x19b2fe thru 0x19cf94 (0x1c97 bytes) 000370[0] 1780533685.025987 mapwrite 0x2aa96b thru 0x2ac403 (0x1a99 bytes) 000373[1] 1780533685.079046 trunc from 0x9d2526 to 0x36883a (0x669ceb bytes) 000377[1] 1780533685.109067 read 0x17f8a7 thru 0x18119d (0x18f7 bytes) 000378[1] 1780533685.148638 read 0x2d88a9 thru 0x2d8e73 (0x5cb bytes) 000379[0] 1780533685.149196 trunc from 0x36883b to 0x6f3d68 (0x38b52e bytes) 000380[0] 1780533685.192613 mapread 0x0ee6bb thru 0x0ef1b4 (0xafa bytes) 000381[1] 1780533685.202102 mapread 0x218e60 thru 0x219e7b (0x101c bytes) 000382[1] 1780533685.253495 read 0x3e58b8 thru 0x3e653b (0xc84 bytes) 000387[1] 1780533685.270928 write 0x35b422 thru 0x35b71a (0x2f9 bytes) 000388[1] 1780533685.316900 write 0x3e6aef thru 0x3e76e9 (0xbfb bytes) 000393[0] 1780533685.363300 mapwrite 0x647326 thru 0x64903f (0x1d1a bytes) 000394[0] 1780533685.406066 trunc from 0x6f3d69 to 0x0b04db (0x64388d bytes) 000395[1] 1780533685.432095 mapwrite 0x3be2cd thru 0x3bf1d3 (0xf07 bytes) 000396[1] 1780533685.480486 read 0x201c62 thru 0x202426 (0x7c5 bytes) 000397[0] 1780533685.534119 trunc from 0x3bf1d4 to 0x227427 (0x197dac bytes) 000398[0] 1780533685.556381 read 0x185e46 thru 0x187ce5 (0x1ea0 bytes) 000401[0] 1780533685.580660 write 0x0d3f99 thru 0x0d5d34 (0x1d9c bytes) 000403[0] 1780533685.593920 mapwrite 0x0f84c4 thru 0x0f8824 (0x361 bytes) 000406[1] 1780533685.627088 mapread 0x16fb1b thru 0x17001c (0x502 bytes) 000407[1] 1780533685.654425 read 0x06ffaa thru 0x070053 (0xaa bytes) 000408[0] 1780533685.678970 trunc from 0x227428 to 0x0b9c16 (0x16d811 bytes) 000410[0] 1780533685.704774 mapwrite 0x47ddb1 thru 0x47f003 (0x1253 bytes) 000411[0] 1780533685.772175 trunc from 0x47f004 to 0x38ed41 (0xf02c2 bytes) 000412[0] 1780533685.787188 mapwrite 0x5a4f67 thru 0x5a6167 (0x1201 bytes) 000414[0] 1780533685.851662 read 0x55d44c thru 0x55d812 (0x3c7 bytes) 000416[1] 1780533685.852662 write 0x7965e9 thru 0x797558 (0xf70 bytes) 000418[1] 1780533685.872533 mapwrite 0x324524 thru 0x3254db (0xfb8 bytes) 000420[1] 1780533685.953101 mapwrite 0x41ac86 thru 0x41c055 (0x13d0 bytes) 000423[1] 1780533685.983678 trunc from 0x797559 to 0x318986 (0x47ebd2 bytes) 000429[0] 1780533686.051822 mapwrite 0x810921 thru 0x8122c0 (0x19a0 bytes) 000434[0] 1780533686.125988 mapwrite 0x2f8f20 thru 0x2fa21e (0x12ff bytes) 000435[0] 1780533686.156826 mapread 0x7202d4 thru 0x72207a (0x1da7 bytes) 000436[0] 1780533686.170429 trunc from 0x8122c1 to 0x805ee7 (0xc3d9 bytes) 000437[1] 1780533686.181939 mapread 0x418ebe thru 0x419cb1 (0xdf4 bytes) 000439[0] 1780533686.207568 mapwrite 0x9d0ec2 thru 0x9d12af (0x3ee bytes) 000440[1] 1780533686.247199 mapread 0x85e542 thru 0x860090 (0x1b4f bytes) 000446[1] 1780533686.276253 trunc from 0x9d12b0 to 0x0af21f (0x922090 bytes) 000447[0] 1780533686.295102 mapread 0x0184e8 thru 0x018f72 (0xa8b bytes) 000448[1] 1780533686.307586 mapread 0x02baba thru 0x02c1d3 (0x71a bytes) 000450[1] 1780533686.323344 mapwrite 0x41e857 thru 0x41f637 (0xde1 bytes) 000453[1] 1780533686.382709 trunc from 0x41f638 to 0x1554a3 (0x2ca194 bytes) 000454[1] 1780533686.392796 mapwrite 0x32513c thru 0x326a21 (0x18e6 bytes) 000455[0] 1780533686.427549 read 0x26a469 thru 0x26b9e5 (0x157d bytes) 000457[0] 1780533686.453588 write 0x8c96f0 thru 0x8ca640 (0xf51 bytes) 000458[1] 1780533686.469074 write 0x857dc3 thru 0x85928a (0x14c8 bytes) 000460[1] 1780533686.498037 read 0x47c611 thru 0x47c8ba (0x2aa bytes) 000462[0] 1780533686.519142 mapwrite 0x28821a thru 0x2893ad (0x1194 bytes) 000463[1] 1780533686.575104 write 0x127901 thru 0x1290dc (0x17dc bytes) 000466[0] 1780533686.604113 write 0x48bc48 thru 0x48d1d0 (0x1589 bytes) 000469[0] 1780533686.638265 trunc from 0x8ca641 to 0x13dc07 (0x78ca39 bytes) 000470[1] 1780533686.668237 trunc from 0x13dc08 to 0x617996 (0x4d9d8f bytes) 000471[1] 1780533686.688658 trunc from 0x617997 to 0x42c997 (0x1eafff bytes) 000473[0] 1780533686.706902 mapwrite 0x8ccb62 thru 0x8ce4e7 (0x1986 bytes) 000477[1] 1780533686.758641 read 0x5ce0f4 thru 0x5cf6a4 (0x15b1 bytes) 000478[1] 1780533686.767827 read 0x0c1d7b thru 0x0c34dc (0x1762 bytes) 000479[1] 1780533686.775960 mapread 0x2b7a19 thru 0x2b7c63 (0x24b bytes) 000481[0] 1780533686.779306 write 0x91b27a thru 0x91b9df (0x766 bytes) 000483[0] 1780533686.784620 mapwrite 0x41b06a thru 0x41b12b (0xc2 bytes) 000484[1] 1780533686.808927 mapwrite 0x9805a3 thru 0x9819e0 (0x143e bytes) 000485[1] 1780533686.848229 mapread 0x77d753 thru 0x77dcdb (0x589 bytes) 000486[1] 1780533686.861817 trunc from 0x9819e1 to 0x50d1c2 (0x47481e bytes) 000488[0] 1780533686.883533 mapread 0x16b09a thru 0x16cd77 (0x1cde bytes) 000490[0] 1780533686.918920 read 0x46cb49 thru 0x46d74f (0xc07 bytes) 000493[0] 1780533686.931684 mapread 0x285beb thru 0x287640 (0x1a56 bytes) 000494[0] 1780533686.943295 read 0x180cb5 thru 0x1821bd (0x1509 bytes) 000495[1] 1780533686.947199 mapwrite 0x73eddd thru 0x73ede8 (0xc bytes) 000497[1] 1780533687.015505 mapread 0x307792 thru 0x3096ee (0x1f5d bytes) 000500[1] 1780533687.046195 read 0x1d1a88 thru 0x1d277b (0xcf4 bytes) All operations completed A-OK! PASS 16b (9s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 20:41:28 (1780533688) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 20:41:30 (1780533690) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.135972 s, 61.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.11975 s, 70.1 MB/s ldlm.namespaces.MGC192.168.203.139@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff925845219000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff92584899c000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff925845219000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff925845219000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.06512 s, 20.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.44853 s, 72.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.03384 s, 101 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.4124 s, 43.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.75226 s, 27.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.49663 s, 70.1 MB/s PASS 16d (21s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 20:41:51 (1780533711) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.151411 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411913 s, 25.5 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 20:41:54 (1780533714) TIMEOUT PASS 16f (24s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 20:42:18 (1780533738) TIMEOUT PASS 16g (23s) == sanityn test 16h: mmap read after truncate file ======= 20:42:41 (1780533761) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (3s) == sanityn test 16i: read after truncate file ============ 20:42:44 (1780533764) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (3s) == sanityn test 16j: race dio with buffered i/o ========== 20:42:47 (1780533767) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.895659 s, 29.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.5257 s, 49.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.05012 s, 25.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.711325 s, 36.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.94031 s, 27.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.885126 s, 29.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.30124 s, 20.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.562924 s, 46.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.730629 s, 35.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.25261 s, 20.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.30224 s, 20.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.971123 s, 27.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.05657 s, 24.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.27154 s, 20.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.34194 s, 19.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.42091 s, 18.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.69873 s, 15.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.71922 s, 15.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.74611 s, 15.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (17s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 20:43:04 (1780533784) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! Wed Jun 3 20:43:20 EDT 2026 + fsx 2 done: rc=0 Wed Jun 3 20:43:20 EDT 2026 + fsx 3 done: rc=0 Wed Jun 3 20:43:20 EDT 2026 + fsx 4 done: rc=0 Wed Jun 3 20:43:20 EDT 2026 + fsx 5 done: rc=0 Wed Jun 3 20:43:20 EDT 2026 PASS 16k (20s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 20:43:24 (1780533804) fail_loc=0x8000030a PASS 17 (6s) == sanityn test 18: mmap sanity check =========================================================================================== 20:43:30 (1780533810) mmap test1: basic mmap operation (PASS, 0.06352s) mmap test2: MAP_PRIVATE not write back (PASS, 0.045559s) mmap test3: concurrent mmap ops on two nodes (PASS, 6.5562s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.2332s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.07405s) mmap test6: check mmap write/read content on two nodes (PASS, 0.095161s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.55095s) mmap test11: multi-thread shared mmap (PASS, 0.58333s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0417s) PASS 18 (17s) == sanityn test 19: test concurrent uncached read races ========================================================================= 20:43:47 (1780533827) oleg339-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 20:43:49 (1780533829) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 20:43:52 (1780533832) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 20:43:55 (1780533835) now is 1780533836 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 new atime is 1780533897 PASS 23 (65s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 20:45:00 (1780533900) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.0M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (3s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 20:45:03 (1780533903) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 20:45:06 (1780533906) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (3s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 20:45:09 (1780533909) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 20:45:11 (1780533911) utime: good mknod times 1780533910 <= 1780533911 <= 1780533911 for /mnt/lustre/f26a utime: good mknod times 1780533910 <= 1780533911 <= 1780533911 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 20:45:14 (1780533914) PASS 26b (5s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 20:45:19 (1780533919) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-03 20:45:20.000000000 -0400 Birth: 2026-06-03 20:45:20.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-03 20:45:20.000000000 -0400 Birth: 2026-06-03 20:45:20.000000000 -0400 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 20:45:24 (1780533924) 0 PASS 30 (5s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 20:45:29 (1780533929) fail_loc=0x314 PASS 31a (4s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 20:45:33 (1780533933) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg339-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.139@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (14s) == sanityn test 31r: open-rename(replace) race =========== 20:45:47 (1780533947) -rw-r--r-- 1 root root 0 Jun 3 20:45 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 20:45:53 (1780533953) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42983 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 20:45:56 (1780533956) PASS 31t (3s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 20:46:00 (1780533960) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 20:46:02 (1780533962) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 20:46:03 (1780533963) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 20:46:05 (1780533965) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 20:46:06 (1780533966) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff925845219000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff925845219000.ost_server_uuid in IDLE state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff925845219000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff925845219000.ost_server_uuid in FULL state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff925845219000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff925845219000.ost_server_uuid in IDLE state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff925845219000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff925845219000.ost_server_uuid in FULL state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff925845219000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff925845219000.ost_server_uuid in IDLE state after 0 sec oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff925845219000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff925845219000.ost_server_uuid in FULL state after 0 sec PASS 34 (97s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 20:47:43 (1780534063) fail_loc=0x80000317 Race attempt 0 Wait for 49032 49165 for 60 sec... - open/close 1611 (time 1780534076.03 total 10.00 last 161.06) - open/close 3358 (time 1780534086.03 total 20.01 last 174.62) total: 4000 open/close in 24.10 seconds: 166.01 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.139@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.203.139@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (65s) == sanityn test 36: handle ESTALE/open-unlink correctly == 20:48:48 (1780534128) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg339-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.516175 s, 102 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.446108 s, 118 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.451836 s, 116 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.491237 s, 107 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.472142 s, 111 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.425385 s, 123 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.483932 s, 108 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.442525 s, 118 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.457637 s, 115 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.495452 s, 106 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.456443 s, 115 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7497728) after_dd(7446528) after(7497728) PASS 36 (235s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 20:52:43 (1780534363) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 - create 5197 (time 1780534374.72 total 10.00 last 519.62) total: 10000 create in 19.85 seconds: 503.82 ops/second PASS 37 (23s) == sanityn test 39a: file mtime does not change after rename ========================================================== 20:53:07 (1780534387) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 20:53:09 (1780534389) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 20:53:12 (1780534392) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 20:53:15 (1780534395) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 20:53:18 (1780534398) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (8s) == sanityn test 40c: pdirops: link and others ======================================================================== 20:53:26 (1780534406) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (8s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 20:53:34 (1780534414) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (8s) == sanityn test 40e: pdirops: rename and others ======================================================================== 20:53:42 (1780534422) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 20:53:50 (1780534430) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (5s) == sanityn test 41b: pdirops: create vs create ======================================================================== 20:53:55 (1780534435) open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 20:54:00 (1780534440) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (5s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 20:54:05 (1780534445) Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 20:54:10 (1780534450) Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 20:54:15 (1780534455) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 20:54:20 (1780534460) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 20:54:25 (1780534465) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (5s) == sanityn test 41i: reint_open: create vs create ======== 20:54:30 (1780534470) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (714s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 21:06:24 (1780535184) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 21:06:29 (1780535189) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 21:06:34 (1780535194) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (5s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 21:06:39 (1780535199) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 21:06:44 (1780535204) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 21:06:49 (1780535209) Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 21:06:54 (1780535214) Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 21:06:59 (1780535219) Conflict PASS 42h (5s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 21:07:04 (1780535224) PASS 43a (16s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 21:07:20 (1780535240) Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 21:07:25 (1780535245) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 21:07:30 (1780535250) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 21:07:35 (1780535255) Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 21:07:40 (1780535260) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 21:07:45 (1780535265) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 21:07:50 (1780535270) Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 21:07:55 (1780535275) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 21:07:56 (1780535276) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (36s) == sanityn test 43k: unlink vs create ==================== 21:08:32 (1780535312) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (408s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 21:15:20 (1780535720) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Jun 3 21:15:24 EDT 2026 PASS 44a (6s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 21:15:26 (1780535726) open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 21:15:31 (1780535731) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 21:15:36 (1780535736) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 21:15:41 (1780535741) Conflict PASS 44e (6s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 21:15:47 (1780535747) Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 21:15:52 (1780535752) Conflict PASS 44g (5s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 21:15:57 (1780535757) Conflict PASS 44h (5s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 21:16:02 (1780535762) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 21:16:03 (1780535763) PASS 45a (26s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 21:16:29 (1780535789) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 21:16:34 (1780535794) Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 21:16:39 (1780535799) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 21:16:44 (1780535804) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 21:16:49 (1780535809) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (5s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 21:16:54 (1780535814) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 21:16:59 (1780535819) Conflict PASS 45h (4s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 21:17:03 (1780535823) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 21:17:04 (1780535824) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (429s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 21:24:13 (1780536253) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 21:24:18 (1780536258) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 21:24:23 (1780536263) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (4s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 21:24:27 (1780536267) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 21:24:32 (1780536272) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 21:24:37 (1780536277) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 21:24:42 (1780536282) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 21:24:47 (1780536287) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 21:24:52 (1780536292) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 21:24:53 (1780536293) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (0s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 21:24:53 (1780536293) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 21:24:54 (1780536294) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 21:24:55 (1780536295) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 21:24:56 (1780536296) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 21:24:57 (1780536297) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 21:24:58 (1780536298) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 21:24:59 (1780536299) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00437246 s, 2.3 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 21:25:06 (1780536306) 0+1 records in 0+1 records out 158 bytes copied, 0.00368432 s, 42.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (217341) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 21:25:10 (1780536310) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00279929 s, 366 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181866 s, 57.7 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 21:25:24 (1780536324) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00418285 s, 245 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 21:25:31 (1780536331) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014219 s, 73.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 21:25:34 (1780536334) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 21:25:38 (1780536338) llite.lustre-ffff925845219000.xattr_cache=1 llite.lustre-ffff92584899c000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 21:25:38.000000000 -0400 Modify: 2026-06-03 21:25:38.000000000 -0400 Change: 2026-06-03 21:25:38.000000000 -0400 Birth: 2026-06-03 21:25:39.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 21:25:44.000000000 -0400 Modify: 2026-06-03 21:25:44.000000000 -0400 Change: 2026-06-03 21:25:44.000000000 -0400 Birth: 2026-06-03 21:25:44.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 21:25:50.000000000 -0400 Modify: 2026-06-03 21:25:50.000000000 -0400 Change: 2026-06-03 21:25:50.000000000 -0400 Birth: 2026-06-03 21:25:50.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 21:25:50.000000000 -0400 Modify: 2026-06-03 21:25:50.000000000 -0400 Change: 2026-06-03 21:25:50.000000000 -0400 Birth: 2026-06-03 21:25:50.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 21:25:55.000000000 -0400 Modify: 2026-06-03 21:25:55.000000000 -0400 Change: 2026-06-03 21:25:55.000000000 -0400 Birth: 2026-06-03 21:25:55.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 21:25:55.000000000 -0400 Modify: 2026-06-03 21:25:55.000000000 -0400 Change: 2026-06-03 21:25:55.000000000 -0400 Birth: 2026-06-03 21:25:55.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 21:26:02 (1780536362) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 21:26:10 (1780536370) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 21:26:18 (1780536378) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (14s) == sanityn test 55d: rename file vs link ================= 21:26:32 (1780536392) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (11s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 21:26:43 (1780536403) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 21:26:44 (1780536404) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 21:26:58 (1780536418) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 21:27:12 (1780536432) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1780536433 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.969 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (34s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 21:27:46 (1780536466) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1780536467 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.21248 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (42s) == sanityn test 60: Verify data_version behaviour ======== 21:28:28 (1780536508) PASS 60 (4s) == sanityn test 70a: cd directory && rm directory ======== 21:28:32 (1780536512) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 21:28:35 (1780536515) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 21:28:38 (1780536518) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00346418 s, 18.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 21:28:39 (1780536519) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00415936 s, 15.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 21:28:40 (1780536520) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 21:28:42 (1780536522) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00570835 s, 11.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 21:28:43 (1780536523) llite.lustre-ffff925845219000.xattr_cache=1 llite.lustre-ffff92584899c000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 21:28:45 (1780536525) llite.lustre-ffff925845219000.xattr_cache=1 llite.lustre-ffff92584899c000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 21:28:48 (1780536528) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00584769 s, 175 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00463716 s, 221 kB/s 230107: get lock2 230109: get lock1 230107: done 230107: get lock1 230109: done 230109: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 230107: cannot get lock1: Resource deadlock avoided 230109: done 230109: exit rc=0 230107: done 230107: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 21:28:54 (1780536534) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0192253 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103447 s, 101 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 230689 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 21:29:02 (1780536542) opening files: ........ closing files: ........ PASS 76 (22s) == sanityn test 77a: check FIFO NRS policy =============== 21:29:24 (1780536564) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.163212 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372557 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684956 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786351 s, 13.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577572 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067053 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606782 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713498 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586816 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747604 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08293 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900934 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692281 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769237 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911306 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122389 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158476 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116999 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119549 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240142 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156084 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181784 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220119 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152927 s, 68.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160407 s, 65.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213593 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378664 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225509 s, 46.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014684 s, 71.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118527 s, 88.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716746 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395693 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151717 s, 69.1 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 21:29:28 (1780536568) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154736 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301807 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362499 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745306 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850022 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113763 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109273 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101883 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112044 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122247 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10038 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111485 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132351 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116402 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104356 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124471 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112463 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126491 s, 82.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141578 s, 74.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185694 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0065321 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183966 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147633 s, 71.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405712 s, 258 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268289 s, 39.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204011 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0030212 s, 347 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150931 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00475635 s, 220 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018514 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043009 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397041 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00553461 s, 189 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.15059 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240455 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356756 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588886 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629807 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730416 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626187 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072567 s, 14.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618065 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739858 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12009 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1272 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124712 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13934 s, 7.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130597 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251282 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149353 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022474 s, 46.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234695 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226207 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325032 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178877 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149567 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00920442 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173387 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160492 s, 65.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110873 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101661 s, 103 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174145 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108119 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00802678 s, 131 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400739 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00820003 s, 128 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 21:29:32 (1780536572) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.121735 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420075 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680346 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635793 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702739 s, 14.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929153 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966974 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966977 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102945 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10887 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108932 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902691 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966409 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794843 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100554 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109866 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929069 s, 11.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148433 s, 70.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178923 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165028 s, 63.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171709 s, 61.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00853711 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116442 s, 90.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134659 s, 77.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190394 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024791 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021507 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00735162 s, 143 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111053 s, 94.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00258104 s, 406 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00698404 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826462 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547609 s, 191 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.138507 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03557 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571318 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857997 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962456 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106056 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110698 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116099 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124973 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14043 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124642 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146746 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146905 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179981 s, 5.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17343 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147641 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164991 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198366 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037474 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316076 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303024 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147719 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234158 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039554 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434989 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00624893 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300036 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293752 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395365 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00655801 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357988 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430675 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00465928 s, 225 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.162056 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274948 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445276 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069002 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689271 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600834 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886137 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858778 s, 12.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823141 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992997 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760671 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982308 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799437 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107262 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823268 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900732 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997839 s, 10.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181072 s, 57.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135038 s, 77.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160871 s, 65.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216774 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182572 s, 57.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00415132 s, 253 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011431 s, 91.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02476 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191721 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210019 s, 49.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203199 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100316 s, 105 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145886 s, 71.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392328 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016751 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00497478 s, 211 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 21:29:38 (1780536578) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.18423 s, 91.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474032 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538935 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657344 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641789 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775965 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806403 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963714 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084555 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940234 s, 11.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845426 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108334 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103624 s, 10.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100916 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110351 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111487 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126788 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105053 s, 99.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199162 s, 52.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197527 s, 53.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213336 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189391 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168189 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123269 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240382 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100861 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385251 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182196 s, 57.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157111 s, 66.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00631535 s, 166 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124576 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00834496 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247918 s, 423 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.211016 s, 79.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367768 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448362 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424284 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536868 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628256 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718174 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605696 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432053 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062603 s, 16.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830876 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906445 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901205 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107033 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106543 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119188 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111975 s, 9.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108919 s, 96.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160934 s, 65.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178923 s, 58.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195442 s, 53.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157323 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141858 s, 73.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00563386 s, 186 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165404 s, 63.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130523 s, 80.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222833 s, 47.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0047242 s, 222 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00683272 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040228 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00369256 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109786 s, 95.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00189305 s, 554 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.197121 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292781 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643842 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757107 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616043 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882039 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091958 s, 11.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870302 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982248 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900097 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102494 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101299 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109897 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101524 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118692 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116644 s, 9.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972976 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126364 s, 83.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139801 s, 75.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185785 s, 56.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178193 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017039 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160569 s, 65.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02283 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121072 s, 86.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259406 s, 40.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197339 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00959974 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235621 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00532785 s, 197 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00691067 s, 152 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00459008 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00972407 s, 108 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 21:29:44 (1780536584) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.39@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246818 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034704 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409332 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225241 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118747 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130727 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116439 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120068 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116098 s, 9.0 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11746 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112062 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126898 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122308 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134401 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117614 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121025 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131058 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213957 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219429 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218333 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217484 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207141 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212132 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211066 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216198 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219159 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223143 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215233 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219435 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228645 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223158 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21832 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22227 s, 4.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.183345 s, 91.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719243 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681662 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465596 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740475 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820192 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585516 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883532 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848742 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739652 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744722 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820488 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889765 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846733 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708943 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704283 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721344 s, 14.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692372 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076393 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617382 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726588 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694582 s, 15.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809019 s, 13.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780867 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743962 s, 14.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814984 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791888 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875449 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727383 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758261 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799283 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784892 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789577 s, 13.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.162228 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510425 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465424 s, 22.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082974 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103014 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109791 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115115 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120843 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139401 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148199 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137198 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138604 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133473 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149898 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151703 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141771 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147936 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300664 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00331934 s, 316 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00325796 s, 322 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359672 s, 292 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291308 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00512089 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400247 s, 262 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0041482 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316629 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00547237 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382054 s, 27.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275459 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369681 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00265687 s, 395 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405461 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00225247 s, 466 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.199106 s, 84.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505109 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548329 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706034 s, 14.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773407 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134507 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878127 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868539 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880026 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138719 s, 7.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146079 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102071 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100699 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153733 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095962 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150447 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135088 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127527 s, 82.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022336 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268594 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268935 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126951 s, 82.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198468 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272659 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00782308 s, 134 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319198 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133339 s, 78.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027385 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246039 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013402 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261057 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017089 s, 61.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00414153 s, 253 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (10s) == sanityn test 77f: check TBF JobID nrs policy ========== 21:29:54 (1780536594) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.200489 s, 83.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288177 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566035 s, 18.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638385 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522694 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520689 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489577 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733337 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798307 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752659 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104278 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830255 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121349 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105774 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107064 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102735 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121843 s, 8.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110601 s, 94.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169235 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158184 s, 66.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113319 s, 92.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217149 s, 48.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213363 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193039 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216673 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264861 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390244 s, 269 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401226 s, 261 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286019 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217756 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032521 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00365971 s, 287 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252287 s, 41.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.219255 s, 76.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031786 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577077 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658842 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855198 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963088 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846711 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798866 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855443 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783977 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12195 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978453 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110951 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10436 s, 10.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100645 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118379 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108 s, 9.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0087971 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140314 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121535 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424004 s, 247 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016674 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00365804 s, 287 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173834 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113746 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219029 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103457 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103702 s, 101 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108285 s, 96.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373507 s, 281 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128066 s, 81.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162908 s, 64.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00883145 s, 119 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.17294 s, 97.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239512 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488206 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103009 s, 10.2 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800484 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740025 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102618 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104519 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972167 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106768 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917364 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105441 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860501 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941892 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113359 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116764 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113468 s, 9.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116515 s, 90.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149906 s, 69.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150514 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018763 s, 55.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018177 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295462 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989483 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00917206 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131153 s, 80.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00725048 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177008 s, 59.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0041799 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013818 s, 75.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189538 s, 55.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0065108 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155819 s, 67.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.237933 s, 70.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372375 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894174 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870975 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103915 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946328 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878819 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102536 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106051 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880198 s, 11.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972854 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102346 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943181 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104898 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111461 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111749 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff92584899c000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff92584899c000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104788 s, 10.0 MB/s execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100139 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117969 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169312 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163634 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363961 s, 288 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013947 s, 75.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199556 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00842076 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213078 s, 49.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00760905 s, 138 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00414634 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193788 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00342698 s, 306 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0107924 s, 97.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00370149 s, 283 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00356746 s, 294 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (9s) == sanityn test 77g: Change TBF type directly ============ 21:30:03 (1780536603) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 21:30:07 (1780536607) oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 PASS 77h (5s) == sanityn test 77i: Change rank of TBF rule ============= 21:30:12 (1780536612) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (10s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 21:30:22 (1780536622) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.185378 s, 90.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201288 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383364 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107577 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105223 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872367 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105823 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102416 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910204 s, 11.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857925 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105976 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114144 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124941 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962355 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106806 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111707 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104553 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237913 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203195 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116755 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209364 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00414997 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285245 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00573503 s, 183 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394373 s, 266 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173772 s, 60.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263128 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243372 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279629 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169882 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010814 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00304623 s, 344 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147941 s, 70.9 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97814 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4394 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (48s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 21:31:10 (1780536670) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24091 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759476 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698262 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915621 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733921 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084139 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905201 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879274 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743577 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133114 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133328 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153498 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118534 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122222 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138665 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146426 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141796 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192461 s, 54.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258696 s, 40.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267281 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195764 s, 53.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246077 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101252 s, 104 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275948 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201557 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019357 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100539 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374379 s, 280 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100141 s, 105 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00917012 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00464462 s, 226 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306406 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00621683 s, 169 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6949 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9175 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22511 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607788 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722537 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821322 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077593 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744254 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910425 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792923 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072149 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151988 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151991 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151195 s, 6.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156628 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133814 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14755 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148254 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17625 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141443 s, 74.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177381 s, 59.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020333 s, 51.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020275 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196668 s, 53.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185752 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039661 s, 264 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172703 s, 60.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160192 s, 65.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221234 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393305 s, 267 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388365 s, 270 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264267 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00525249 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121444 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00182248 s, 575 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8362 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9236 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (139s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 21:33:29 (1780536809) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23566 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089746 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640785 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082235 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954769 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984337 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870215 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732603 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900393 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137499 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137699 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138877 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136247 s, 7.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119556 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123945 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1631 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14535 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139622 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222386 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235534 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0060856 s, 172 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207353 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272341 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0062745 s, 167 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190296 s, 55.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152626 s, 68.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199649 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345646 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125659 s, 83.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00873724 s, 120 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00651353 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903424 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00450632 s, 233 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.846 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9163 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23438 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105036 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114608 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152361 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153024 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133007 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12201 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135264 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153064 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143195 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135596 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139171 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167643 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143913 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161548 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163434 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164651 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198343 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214768 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196476 s, 53.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024821 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181037 s, 57.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105918 s, 99.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250469 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273697 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255248 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219332 s, 47.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00964705 s, 109 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109037 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392562 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130593 s, 80.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.008491 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00463226 s, 226 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8297 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9709 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (139s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 21:35:48 (1780536948) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.247422 s, 67.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404237 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591274 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554655 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810747 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659298 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935027 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684412 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849713 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101735 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10047 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956334 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910993 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954639 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912691 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102735 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102312 s, 10.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229447 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211597 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173994 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192961 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148972 s, 70.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204507 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207429 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014747 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220954 s, 47.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187933 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00311034 s, 337 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360568 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112274 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366465 s, 286 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00337338 s, 311 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00155246 s, 675 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92946 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73739 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.39@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.39@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.122746 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221727 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730897 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774831 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765667 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087891 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862751 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733407 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914741 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916453 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895501 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154775 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170507 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235712 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220289 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253808 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281299 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105873 s, 99.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124724 s, 84.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00882844 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177442 s, 59.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249012 s, 42.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00722407 s, 145 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170131 s, 61.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229745 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00971241 s, 108 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384992 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01168 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238526 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267898 s, 39.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244681 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00542865 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00729266 s, 144 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95226 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74823 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.39@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130627 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352248 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511061 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086303 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780258 s, 13.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916892 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781775 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903072 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111602 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101823 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107636 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108564 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097275 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993384 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101878 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107721 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126173 s, 8.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00866792 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160098 s, 65.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126156 s, 83.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177069 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141499 s, 74.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136444 s, 76.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036368 s, 288 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184503 s, 56.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113087 s, 92.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173926 s, 60.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240854 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00364628 s, 288 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00757849 s, 138 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460393 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409759 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122485 s, 85.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94769 s, 21.2 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88218 s, 21.5 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.126372 s, 133 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258915 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436648 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364041 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705246 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770832 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816378 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704909 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100611 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847214 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771397 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104114 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910584 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898557 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102449 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127889 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105421 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170934 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164006 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162645 s, 64.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231832 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204165 s, 51.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247133 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255056 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00926075 s, 113 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377872 s, 277 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290022 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00496114 s, 211 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291297 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118798 s, 88.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123564 s, 84.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103867 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00214167 s, 490 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77689 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73609 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94797 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74185 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.864712 s, 19.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278562 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421736 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609236 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757622 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649717 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592383 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642784 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609782 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145439 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168795 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181568 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175815 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173143 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182662 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19362 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200201 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145114 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176769 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159674 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023805 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017431 s, 60.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374542 s, 280 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020821 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246323 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022323 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222649 s, 47.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00507622 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016447 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207497 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426834 s, 246 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214726 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177348 s, 59.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6832 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4328 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.124058 s, 135 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034784 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286613 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101722 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871256 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099517 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911705 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932076 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920411 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100825 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929029 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106491 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110657 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127304 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099544 s, 10.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126873 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108211 s, 9.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148558 s, 70.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135759 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140654 s, 74.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132055 s, 79.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020192 s, 51.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156021 s, 67.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126647 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00891126 s, 118 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215974 s, 48.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242587 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00467601 s, 224 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287531 s, 36.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00584969 s, 179 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258136 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00645174 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00457979 s, 229 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94669 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72898 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.169999 s, 98.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479226 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041954 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105091 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13737 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129114 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137552 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146137 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167276 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155229 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167823 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192412 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18054 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186048 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19044 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298358 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310513 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177205 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165162 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192766 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145538 s, 72.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230498 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202508 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216476 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245176 s, 42.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213609 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246826 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0050108 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00970499 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211027 s, 49.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00994604 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00311227 s, 337 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183196 s, 57.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78687 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74752 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76854 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.742 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (359s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 21:41:47 (1780537307) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.273936 s, 61.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556245 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934458 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100388 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119938 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101619 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984911 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156752 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148102 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139857 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131449 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1597 s, 6.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186381 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151946 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166117 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196569 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16445 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247729 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288534 s, 36.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306109 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309114 s, 33.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306752 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336171 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029055 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00894826 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120907 s, 86.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198608 s, 52.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325777 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188776 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229124 s, 45.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129476 s, 81.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331866 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222713 s, 47.1 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9638 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74678 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (42s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 21:42:29 (1780537349) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.39@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.39@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.325648 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512421 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411972 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154999 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130932 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170102 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162116 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138803 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166867 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161064 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167125 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166635 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176933 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17446 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175999 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241327 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23009 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242362 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346899 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326963 s, 32.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335487 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031106 s, 33.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162436 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016144 s, 65.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102033 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260262 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161099 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169605 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216723 s, 48.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126572 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177251 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269294 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141648 s, 74.0 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94823 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74865 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (39s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 21:43:09 (1780537389) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.39@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147055 s, 114 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020504 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510566 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316823 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497649 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497753 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562357 s, 18.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951137 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11107 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102864 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993301 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105943 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093457 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116194 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106767 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104803 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114923 s, 9.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119453 s, 87.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170933 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194564 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125634 s, 83.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169759 s, 61.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018775 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242318 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216529 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237722 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037924 s, 276 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02204 s, 47.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00552855 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217254 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241859 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236139 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254565 s, 41.2 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95788 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97692 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (36s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 21:43:45 (1780537425) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.479125 s, 35.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509787 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581225 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076541 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691648 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112768 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914623 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881222 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112132 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11675 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876552 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102609 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104335 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101777 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886448 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14247 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153773 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206875 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261544 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231328 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252853 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259912 s, 40.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249983 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00682654 s, 154 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237859 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120776 s, 86.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220276 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00377974 s, 277 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282966 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124555 s, 84.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277523 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421431 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554051 s, 189 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82248 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73643 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93736 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7466 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (77s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 21:45:02 (1780537502) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.875345 s, 19.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263797 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539263 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058485 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697018 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647938 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671772 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806876 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730107 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13297 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145752 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155121 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17974 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192515 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195745 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163256 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183594 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208998 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112019 s, 93.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023872 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227924 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214352 s, 48.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248416 s, 42.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208387 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00357628 s, 293 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213682 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00395575 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148256 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027499 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399222 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374805 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254493 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00173815 s, 603 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5571 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9025 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (67s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 21:46:09 (1780537569) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.154439 s, 109 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360109 s, 29.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267655 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554851 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642057 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588142 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670114 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847006 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844287 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972111 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783967 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681499 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854263 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969986 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108177 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970753 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965348 s, 10.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015942 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189779 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213854 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116775 s, 89.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172098 s, 60.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167757 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375334 s, 279 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202955 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175707 s, 59.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206354 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0090263 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00338058 s, 310 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00543679 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00746116 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147391 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00462734 s, 227 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96443 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74985 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg339-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.48039 s, 34.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409494 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356214 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11659 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090332 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119015 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10458 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105783 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10072 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101311 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112392 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128484 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109841 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116302 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123367 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158917 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164648 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172049 s, 60.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118047 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224761 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168169 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164458 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0083857 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024383 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199808 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199859 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00941693 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00559333 s, 187 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142487 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190841 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169782 s, 61.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00588347 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00644158 s, 163 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80495 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74586 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77632 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73415 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (121s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 21:48:10 (1780537690) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:) Stopping client oleg339-client.virtnet /mnt/lustre opts: Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg339-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg339-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.39,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg339-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.39,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg339-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.39,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/64653/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.39,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg339-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1044: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1044: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Starting client oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Started clients oleg339-client.virtnet: 192.168.203.139@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Starting client oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 Started clients oleg339-client.virtnet: 192.168.203.139@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9258488c2800.idle_timeout=debug osc.lustre-OST0000-osc-ffff92586b1e3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9258488c2800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92586b1e3800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.274536 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345027 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466993 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672155 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894565 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781647 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795616 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972743 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759889 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104612 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975951 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105027 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115317 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213203 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227939 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257224 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274304 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185441 s, 56.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283414 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321419 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028988 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189424 s, 55.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220713 s, 47.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231192 s, 45.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142934 s, 73.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212152 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274371 s, 38.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326915 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00602975 s, 174 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00507072 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00553874 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148171 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206759 s, 50.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95047 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97364 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.310021 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815315 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755802 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100726 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106472 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939393 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819822 s, 12.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805276 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994524 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119936 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135033 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164451 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141638 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167712 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193509 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220587 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228726 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032421 s, 32.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400753 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450412 s, 23.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198233 s, 52.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223546 s, 46.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408034 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289845 s, 36.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338542 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213547 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020751 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255428 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232405 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384033 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199499 s, 52.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350392 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131354 s, 79.8 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9522 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74389 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (141s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 21:50:31 (1780537831) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg339-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (9s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 21:50:41 (1780537841) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 21:50:44 (1780537844) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg339-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.77256 s, 278 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg339-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0198 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (52s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 21:51:36 (1780537896) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.175637 s, 95.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409919 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101026 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979633 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10822 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125411 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138768 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128551 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13719 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142717 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1226 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140072 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142367 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137834 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155192 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152215 s, 6.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff92586b1e3800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165407 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176312 s, 59.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173272 s, 60.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207699 s, 50.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167648 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283407 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302184 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156107 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117116 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236965 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103176 s, 102 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347214 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126498 s, 82.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213637 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350373 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288422 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164551 s, 63.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg339-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg339-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94003 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97615 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.1215 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292572 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365817 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059815 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056813 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395077 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641896 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659956 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878845 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105369 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553772 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105907 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937274 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151389 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173666 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219624 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213909 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209792 s, 50.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231731 s, 45.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210898 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027618 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221595 s, 47.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269679 s, 38.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209683 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259502 s, 40.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251549 s, 41.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278374 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290442 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313715 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243798 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261166 s, 40.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300223 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030215 s, 34.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg339-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg339-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92416 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96565 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (66s) == sanityn test 77o: Changing rank should not panic ====== 21:52:42 (1780537962) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 21:52:46 (1780537966) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (41s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 21:53:27 (1780538007) mds.MDS.mdt.nrs_policies=tbf oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg339-client: oleg339-server: ssh exited with exit code 36 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg339-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg339-client: oleg339-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (15s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 21:53:42 (1780538022) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 272156 kill md_thread xargs PASS 77r (39s) == sanityn test 77s: Check TBF LRU shrinker ============== 21:54:21 (1780538061) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (13s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 21:54:34 (1780538074) oleg339-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg339-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg339-client: oleg339-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 21:54:37 (1780538077) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826936186 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 21:54:37.000000000 -0400 Modify: 2026-06-03 21:54:37.000000000 -0400 Change: 2026-06-03 21:54:37.000000000 -0400 Birth: 2026-06-03 21:54:37.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (12s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 21:54:49 (1780538089) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 21:54:51 (1780538091) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (2s) == sanityn test 81a: rename and stat under striped directory ========================================================== 21:54:52 (1780538092) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 21:54:53 (1780538093) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 21:54:54 (1780538094) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 21:54:55 (1780538095) total: 2500 open/close in 5.29 seconds: 472.22 ops/second snapshot_time 1780538123.819501393 secs.nsecs start_time 1780538096.116647363 secs.nsecs elapsed_time 27.702854030 secs.nsecs open 2505 samples [usecs] 78 2071 927532 364123296 close 2504 samples [usecs] 68 1306 244112 26841620 mknod 2501 samples [usecs] 221 2005 828536 292263516 unlink 2500 samples [usecs] 338 24910 2300881 4383403807 mkdir 2 samples [usecs] 851 909 1760 1550482 rmdir 2 samples [usecs] 3252 7638 10890 68914548 rename 2500 samples [usecs] 279 12569 3243815 6153275135 getattr 3760 samples [usecs] 6 12075 108658 152839948 parallel_rename_file 2500 samples [usecs] 279 12569 3243815 6153275135 crossdir_rename 2500 samples [usecs] 279 12569 3243815 6153275135 PASS 81d (31s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 21:55:26 (1780538126) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7577 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7577 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 21:55:28 (1780538128) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 21:55:30 (1780538130) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 21:55:38 (1780538138) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1780538139 Finishing test test1 at 1780538141 PASS 85 (6s) == sanityn test 90: open/create and unlink striped directory ========================================================== 21:55:44 (1780538144) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 21:55:45 (1780538145) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 21:55:47 (1780538147) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 21:55:48 (1780538148) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6341 0x18c5 0x240000400 1 6304 0x18a0 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6342 0x18c6 0x240000400 1 6305 0x18a1 0x280000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 21:55:57 (1780538157) osc.lustre-OST0000-osc-ffff9258488c2800.idle_timeout=0 osc.lustre-OST0000-osc-ffff92586b1e3800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9258488c2800.idle_timeout=0 osc.lustre-OST0001-osc-ffff92586b1e3800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0366823 s, 223 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09105 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6178: 292511 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0110014 s, 372 kB/s osc.lustre-OST0000-osc-ffff9258488c2800.idle_timeout=debug osc.lustre-OST0000-osc-ffff92586b1e3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9258488c2800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92586b1e3800.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 21:56:07 (1780538167) llite.lustre-ffff9258488c2800.fast_read=0 llite.lustre-ffff92586b1e3800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0211109 s, 99.3 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9258488c2800.fast_read=1 llite.lustre-ffff92586b1e3800.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 21:56:22 (1780538182) llite.lustre-ffff9258488c2800.fast_read=0 llite.lustre-ffff92586b1e3800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00190105 s, 6.5 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9258488c2800.fast_read=1 llite.lustre-ffff92586b1e3800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 21:56:30 (1780538190) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 21:56:31 (1780538191) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00180715 s, 2.3 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 21:56:34 (1780538194) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0346795 s, 60.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 21:56:36 (1780538196) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.037977 s, 55.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 21:56:38 (1780538198) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00158153 s, 7.6 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 21:56:40 (1780538200) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00164499 s, 2.5 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 21:56:42 (1780538202) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00161758 s, 2.5 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 21:56:44 (1780538204) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00130231 s, 3.1 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 21:56:48 (1780538208) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1780538208 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1780538211 Finishing test test23 at 1780538211 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 21:56:55 (1780538215) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 21:56:56 (1780538216) fail_loc=0x416 PASS 105 (18s) == sanityn test 106a: Verify the btime via statx() ======= 21:57:14 (1780538234) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 21:57:15 (1780538235) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00241287 s, 424 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938721 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 21:57:15.000000000 -0400 Modify: 2026-06-03 21:57:15.000000000 -0400 Change: 2026-06-03 21:57:15.000000000 -0400 Birth: 2026-06-03 21:57:15.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938721 -rw-r--r-- 1 0 0 1780538235 1780538235 1780538235 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938721 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 21:57:15.000000000 -0400 Modify: 2026-06-03 21:57:15.000000000 -0400 Change: 2026-06-03 21:57:15.000000000 -0400 Birth: 2026-06-03 21:57:15.000000000 -0400 1780538235 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 21:57:17 (1780538237) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 21:57:19 (1780538239) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.104924 s, 99.9 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6311 0x18a7 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 21:57:23 (1780538243) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0888273 s, 118 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6351 0x18cf 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7577 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 21:57:31 (1780538251) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00161608 s, 6.2 MB/s llite.lustre-ffff9258488c2800.hybrid_io=0 llite.lustre-ffff92586b1e3800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.17184 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 21:57:38 (1780538258) 192.168.203.139@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 192.168.203.139@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1044: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1044: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 192.168.203.139@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1044: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1044: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 192.168.203.139@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.139@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg339-client.virtnet /mnt/lustre (opts:) Stopping client oleg339-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1044: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1044: echo: write error: Device or resource busy oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre2 PASS 109 (34s) == sanityn test 110: do not grant another lock on resend ========================================================== 21:58:12 (1780538292) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-03 21:58:12.000000000 -0400 Modify: 2026-06-03 21:58:12.000000000 -0400 Change: 2026-06-03 21:58:12.000000000 -0400 Birth: 2026-06-03 21:58:12.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg339-client.virtnet oleg339-client.virtnet PASS 110 (165s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 22:00:57 (1780538457) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 22:00:59 (1780538459) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 22:01:00 (1780538460) lustre-OST0000-osc-ffff925860f30000 active. lustre-OST0001-osc-ffff925860f30000 active. lustre-OST0000-osc-ffff925860f30000 active. lustre-OST0001-osc-ffff925860f30000 active. MGC192.168.203.139@tcp active. PASS 113 (3s) == sanityn test 114: implicit default LMV inherit ======== 22:01:03 (1780538463) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 22:01:04 (1780538464) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (2s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 22:01:06 (1780538466) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 22:01:07 (1780538467) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00268496 s, 7.6 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0129506 s, 1.6 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (4s) == sanityn test 200: service remains healthy while able to process request ========================================================== 22:01:11 (1780538471) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=8 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.084 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0896 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.087 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0857 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0908 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.097 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1442 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1388 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1491 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1553 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1598 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1546 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2515 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2461 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2495 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2412 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg339-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff925860f30000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff925860f30000.ost_server_uuid in FULL state after 0 sec PASS 200 (68s) cleanup: ====================================================== == sanityn test complete, duration 5176 sec ============== 22:02:20 (1780538540) === sanityn: start cleanup 22:02:21 (1780538541) === Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg339-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 22:03:30 (1780538610) ===