-----============= acceptance-small: sanityn ============----- Fri Sep 4 09:27:08 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg415-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 09:27:31 (1788528451) === Starting client oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 Mount client oleg415-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 Started clients oleg415-client.virtnet: 192.168.204.115@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg415-client.virtnet: executing check_config_client /mnt/lustre oleg415-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg415-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c5dc6121000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c5dd1e9c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c5dc6121000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c5dd1e9c800.idle_timeout=debug disable quota as required oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 09:28:09 (1788528489) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235029 s, 44.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7614] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 09:28:11 (1788528491) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 09:28:19 (1788528499) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 09:28:26 (1788528506) -rw-r--r-- 1 root root 0 Sep 4 09:28 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 09:28:34 (1788528514) -rw-r--r-- 1 root root 0 Sep 4 09:28 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 09:28:40 (1788528520) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 09:28:47 (1788528527) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 09:28:54 (1788528534) total 1 -rwxrwxrwx 1 root root 0 Sep 4 09:28 f2a -rwxrwxrwx 1 root root 0 Sep 4 09:28 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 4 09:28 f2a -rwxrwxrwx 1 root root 0 Sep 4 09:28 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0183115 s, 28.0 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 09:29:02 (1788528542) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 09:29:05 (1788528545) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.390043 s, 5.4 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 09:29:12 (1788528552) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 09:29:20 (1788528560) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 09:29:27 (1788528567) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 09:29:34 (1788528574) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 09:29:41 (1788528581) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 09:29:48 (1788528588) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 09:29:56 (1788528596) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 09:30:04 (1788528604) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724392 s, 14.5 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.084222 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 09:30:11 (1788528611) 1+0 records in 1+0 records out 1 byte copied, 0.0163735 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0123148 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00928042 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0176084 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0109378 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00943272 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.015884 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.025727 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0140638 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0166625 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0129051 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0165279 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 09:30:19 (1788528619) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0335188 s, 91.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0290203 s, 141 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000322554 s, 9.5 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 09:30:26 (1788528626) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 09:30:32 (1788528632) warning: '-runtime' deprecated, use '-t runtime' instead - link 309 (time 1788528645.44 total 10.00 last 30.89) - link 581 (time 1788528655.47 total 20.03 last 27.12) - link 860 (time 1788528665.48 total 30.05 last 27.86) - link 1161 (time 1788528675.50 total 40.06 last 30.05) - link 1464 (time 1788528685.50 total 50.07 last 30.29) - link 1725 (time 1788528695.51 total 60.07 last 26.09) - link 2002 (time 1788528705.51 total 70.08 last 27.69) - link 2285 (time 1788528715.51 total 80.08 last 28.29) - link 2547 (time 1788528725.55 total 90.11 last 26.11) total: 2842 link in 100.02 seconds: 28.41 ops/second using seed 3275450715 running for 100 seconds - stat 5010 (time 1788528647 ; total 11 ; last 11) - stat 9969 (time 1788528658 ; total 22 ; last 11) - stat 10000 (time 1788528658 ; total 22 ; last 0) - stat 15507 (time 1788528669 ; total 33 ; last 11) - stat 20000 (time 1788528678 ; total 42 ; last 9) - stat 24257 (time 1788528689 ; total 53 ; last 11) - stat 26896 (time 1788528700 ; total 64 ; last 11) - stat 29209 (time 1788528711 ; total 75 ; last 11) - stat 30000 (time 1788528714 ; total 78 ; last 3) - stat 32566 (time 1788528725 ; total 89 ; last 11) - stat 35074 (time 1788528736 ; total 100 ; last 11) total: 35075 stats in 100 seconds: 350.750000 stats/second - unlinked 0 (time 1788528686 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2842) error: No such file or directory total: 2841 unlinks in 87 seconds: 32.655174 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21423) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21425) - No such process PASS 12 (147s) == sanityn test 13: test directory page revocation ======= 09:32:59 (1788528779) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 09:33:10 (1788528790) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22609 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 09:33:17 (1788528797) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23188 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 09:33:24 (1788528804) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23767 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 09:33:31 (1788528811) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24349 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 09:33:38 (1788528818) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24931 Terminated $DIR1/$tdir/sleep 60 PASS 14d (11s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 09:33:49 (1788528829) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (53s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 09:34:42 (1788528882) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.895381 s, 9.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.61563 s, 13.6 MB/s Chance of close/open is 1 in 50 Seed set to 8888 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x5407aa 000001[0] 1788528889.463870 trunc from 00000000 to 0x5407a9 (0x5407aa bytes) 000004[0] 1788528889.600585 mapread 0x2a68c3 thru 0x2ae3b0 (0x7aee bytes) 000005[1] 1788528889.644406 mapwrite 0x41d286 thru 0x42875b (0xb4d6 bytes) 000006[0] 1788528889.860851 mapread 0x130fd1 thru 0x137589 (0x65b9 bytes) 000007[0] 1788528889.938726 punch from 0x207517 to 0x2160d9 (0xebc3 bytes) 000008[1] 1788528889.979531 write 0x57d5b9 thru 0x588c5e (0xb6a6 bytes) 000009[1] 1788528890.002829 read 0x4613f7 thru 0x46b3ca (0x9fd4 bytes) 000011[0] 1788528890.079365 write 0x82ed66 thru 0x830b09 (0x1da4 bytes) 000014[1] 1788528890.202452 mapwrite 0x10b3b8 thru 0x11a17d (0xedc6 bytes) 000016[0] 1788528890.436641 mapread 0x3c0dd4 thru 0x3d0a7f (0xfcac bytes) 000018[0] 1788528890.645783 mapread 0x1a10e4 thru 0x1ac9ac (0xb8c9 bytes) 000020[1] 1788528890.693845 punch from 0x5b631d to 0x5b8fc1 (0x2ca5 bytes) 000021[1] 1788528890.734824 read 0x811c90 thru 0x818e73 (0x71e4 bytes) 000023[1] 1788528890.777778 mapwrite 0x69298c thru 0x6a0702 (0xdd77 bytes) 000024[0] 1788528891.070513 mapwrite 0x554266 thru 0x55a17c (0x5f17 bytes) 000025[0] 1788528891.156701 trunc from 0x830b0a to 0x3635e0 (0x4cd529 bytes) 000026[0] 1788528891.249577 read 0x3388bf thru 0x34627a (0xd9bc bytes) 000027[1] 1788528891.267502 punch from 0x1bebc0 to 0x1ca1ea (0xb62b bytes) 000028[1] 1788528891.304814 read 0x33544c thru 0x343474 (0xe029 bytes) 000029[0] 1788528891.415549 write 0x3e070c thru 0x3ed245 (0xcb3a bytes) 000031[1] 1788528891.541463 mapwrite 0x679866 thru 0x67eaec (0x5287 bytes) 000032[0] 1788528892.033838 mapread 0x462f19 thru 0x463565 (0x64d bytes) 000033[1] 1788528892.143709 mapread 0x67c0b1 thru 0x67eaec (0x2a3c bytes) 000035[0] 1788528892.163517 mapread 0x4c5b73 thru 0x4cf962 (0x9df0 bytes) 000037[1] 1788528892.207055 punch from 0x332fa1 to 0x337db4 (0x4e14 bytes) 000038[0] 1788528892.224425 mapwrite 0x3354a5 thru 0x33627c (0xdd8 bytes) 000039[1] 1788528892.410459 read 0x33f0bb thru 0x34708a (0x7fd0 bytes) 000041[0] 1788528892.438968 mapread 0x092eb5 thru 0x09860d (0x5759 bytes) 000042[1] 1788528892.464318 read 0x6005b2 thru 0x606503 (0x5f52 bytes) 000044[1] 1788528892.476664 write 0x9fbfa2 thru 0x9fedb3 (0x2e12 bytes) 000045[1] 1788528892.529808 read 0x2630fc thru 0x272f8c (0xfe91 bytes) 000046[1] 1788528892.565362 mapwrite 0x219cdc thru 0x21aabd (0xde2 bytes) truncating to largest ever: 0x95be14 000048[1] 1788528892.652916 trunc from 0x9fedb4 to 0x95be13 (0xa2fa0 bytes) 000051[1] 1788528892.795677 write 0x8c754e thru 0x8c91c0 (0x1c73 bytes) 000053[0] 1788528892.844542 punch from 0x5a4e63 to 0x5b359f (0xe73d bytes) 000054[0] 1788528892.878373 mapread 0x3bc4cd thru 0x3c32a8 (0x6ddc bytes) 000055[1] 1788528892.931045 mapread 0x083b69 thru 0x089437 (0x58cf bytes) 000057[0] 1788528892.966807 write 0x2f05df thru 0x2f760f (0x7031 bytes) 000058[0] 1788528893.024803 write 0x19afe6 thru 0x19cc3a (0x1c55 bytes) 000059[1] 1788528893.078831 mapread 0x66811e thru 0x668ead (0xd90 bytes) 000061[1] 1788528893.197463 trunc from 0x95be14 to 0x4afded (0x4ac026 bytes) 000062[1] 1788528893.301057 mapwrite 0x66a857 thru 0x66e86b (0x4015 bytes) 000063[1] 1788528893.469939 write 0x8a4108 thru 0x8b3824 (0xf71d bytes) 000064[1] 1788528893.476081 trunc from 0x8b3825 to 0x6eb39b (0x1c8489 bytes) 000065[0] 1788528893.554882 punch from 0x23fe3e to 0x245632 (0x57f5 bytes) 000067[1] 1788528893.667757 write 0x9e02f0 thru 0x9e464f (0x4360 bytes) 000068[1] 1788528893.689466 write 0x467606 thru 0x473656 (0xc051 bytes) 000070[1] 1788528893.737880 read 0x924c5b thru 0x9288a1 (0x3c47 bytes) 000071[0] 1788528893.747717 mapwrite 0x90ae13 thru 0x918313 (0xd501 bytes) 000073[0] 1788528893.900448 punch from 0x6baf7d to 0x6bc7f9 (0x187d bytes) 000074[1] 1788528893.939814 write 0x206970 thru 0x2094e2 (0x2b73 bytes) 000076[0] 1788528894.063789 read 0x9e1faa thru 0x9e464f (0x26a6 bytes) 000077[1] 1788528894.140741 mapread 0x4faeb2 thru 0x4fe1b0 (0x32ff bytes) 000078[1] 1788528894.264782 mapwrite 0x084db5 thru 0x092f1d (0xe169 bytes) 000080[1] 1788528894.429100 punch from 0x3da406 to 0x3da8f4 (0x4ef bytes) 000081[0] 1788528894.474780 write 0x097be4 thru 0x09b64f (0x3a6c bytes) 000085[1] 1788528894.593441 trunc from 0x9e4650 to 0x12edf3 (0x8b585c bytes) 000087[1] 1788528894.753065 mapread 0x0739e8 thru 0x079248 (0x5861 bytes) 000089[1] 1788528894.794822 mapread 0x121b24 thru 0x129a34 (0x7f11 bytes) 000090[1] 1788528894.823709 mapread 0x0cde5c thru 0x0cf7f4 (0x1999 bytes) 000091[1] 1788528894.852148 mapwrite 0x4331a0 thru 0x440ab2 (0xd913 bytes) 000093[0] 1788528895.227518 read 0x4355e3 thru 0x440ab2 (0xb4d0 bytes) 000094[0] 1788528895.273607 write 0x6671c5 thru 0x676d45 (0xfb81 bytes) 000095[0] 1788528895.306006 read 0x650694 thru 0x65c2aa (0xbc17 bytes) 000096[1] 1788528895.358529 write 0x16bdb7 thru 0x176145 (0xa38f bytes) 000097[0] 1788528895.391665 read 0x20b80f thru 0x21a162 (0xe954 bytes) 000099[0] 1788528895.474064 write 0x0195b9 thru 0x028d04 (0xf74c bytes) 000100[1] 1788528895.517009 write 0x32a783 thru 0x335c0b (0xb489 bytes) 000102[1] 1788528895.623741 mapwrite 0x411f6e thru 0x41ccdd (0xad70 bytes) 000103[1] 1788528895.761883 read 0x5dee72 thru 0x5e7d69 (0x8ef8 bytes) 000106[1] 1788528895.809038 trunc from 0x676d46 to 0x2c97a6 (0x3ad59f bytes) 000107[1] 1788528895.902821 mapread 0x0133e3 thru 0x015ef1 (0x2b0f bytes) 000109[1] 1788528895.938769 trunc from 0x2c97a7 to 0x80ea53 (0x5452ad bytes) 000110[0] 1788528896.025804 read 0x5d127a thru 0x5def31 (0xdcb8 bytes) 000111[1] 1788528896.100876 punch from 0x6127cb to 0x61a1af (0x79e5 bytes) 000112[0] 1788528896.129936 read 0x2dd94d thru 0x2dda8d (0x141 bytes) 000114[0] 1788528896.203214 read 0x267a6f thru 0x26eb3c (0x70ce bytes) 000115[0] 1788528896.231724 write 0x099e57 thru 0x0a3b33 (0x9cdd bytes) 000117[0] 1788528896.330780 trunc from 0x80ea54 to 0x85a8f2 (0x4be9f bytes) 000118[0] 1788528896.404421 mapread 0x6701b1 thru 0x67d052 (0xcea2 bytes) 000119[0] 1788528896.438242 mapread 0x00494e thru 0x00d9e3 (0x9096 bytes) 000122[0] 1788528896.481928 read 0x7ed6a1 thru 0x7f3cd0 (0x6630 bytes) 000123[1] 1788528896.546688 mapwrite 0x2b75a5 thru 0x2b8a6c (0x14c8 bytes) 000125[1] 1788528896.736667 mapwrite 0x01c3ce thru 0x01c578 (0x1ab bytes) 000129[1] 1788528896.942217 mapread 0x55e936 thru 0x56cbf8 (0xe2c3 bytes) 000130[0] 1788528897.068672 mapwrite 0x0f20f2 thru 0x0fcd5b (0xac6a bytes) 000131[1] 1788528897.342837 mapread 0x40fa2a thru 0x41e8ad (0xee84 bytes) 000132[0] 1788528897.473605 trunc from 0x85a8f3 to 0x199273 (0x6c167f bytes) 000133[0] 1788528897.513373 trunc from 0x199274 to 0x6545ef (0x4bb37c bytes) 000134[0] 1788528897.557394 trunc from 0x6545f0 to 0x79bf80 (0x147991 bytes) 000136[1] 1788528897.616653 trunc from 0x79bf81 to 0x348e65 (0x45311b bytes) 000137[0] 1788528897.658210 punch from 0x1536a7 to 0x162a02 (0xf35c bytes) 000138[1] 1788528897.692943 trunc from 0x348e66 to 0x09e076 (0x2aadef bytes) 000140[1] 1788528897.732981 write 0x55672a thru 0x55dd4a (0x7621 bytes) 000141[1] 1788528897.763341 mapwrite 0x5fcf89 thru 0x608b54 (0xbbcc bytes) 000143[1] 1788528897.893479 punch from 0x0cfe16 to 0x0dece1 (0xeecc bytes) 000144[0] 1788528897.920245 read 0x02f8a6 thru 0x030c59 (0x13b4 bytes) 000146[1] 1788528897.958784 read 0x2b5d31 thru 0x2bb69d (0x596d bytes) 000147[1] 1788528897.984484 write 0x1b73ca thru 0x1b7b3f (0x776 bytes) 000149[1] 1788528898.048344 write 0x418503 thru 0x41f316 (0x6e14 bytes) 000153[0] 1788528898.102884 punch from 0x464a83 to 0x46c2cb (0x7849 bytes) 000155[0] 1788528898.158907 write 0x5c204e thru 0x5cbceb (0x9c9e bytes) 000157[1] 1788528898.196003 mapwrite 0x6209c9 thru 0x62c91c (0xbf54 bytes) 000161[1] 1788528898.340912 trunc from 0x62c91d to 0x6b88e9 (0x8bfcd bytes) 000164[1] 1788528898.387192 write 0x439368 thru 0x43b27c (0x1f15 bytes) 000165[1] 1788528898.397676 write 0x6da7f6 thru 0x6ddf97 (0x37a2 bytes) 000167[0] 1788528898.434796 write 0x98b6d6 thru 0x99a95d (0xf288 bytes) 000168[1] 1788528898.484633 punch from 0x235fa8 to 0x238748 (0x27a1 bytes) 000169[1] 1788528898.559752 trunc from 0x99a95e to 0x587397 (0x4135c6 bytes) 000170[0] 1788528898.603041 punch from 0x04baf5 to 0x05b6b2 (0xfbbe bytes) 000173[0] 1788528898.706621 punch from 0x5571a0 to 0x5590b9 (0x1f1a bytes) 000174[1] 1788528898.769320 mapread 0x38109a thru 0x38db3f (0xcaa6 bytes) 000175[0] 1788528898.861494 mapread 0x37e183 thru 0x3847b9 (0x6637 bytes) 000176[1] 1788528898.895072 read 0x19c7cc thru 0x1a8b00 (0xc335 bytes) 000179[1] 1788528898.957377 punch from 0x1cba98 to 0x1d2978 (0x6ee1 bytes) 000180[0] 1788528899.029522 read 0x4a23e5 thru 0x4a4f75 (0x2b91 bytes) 000183[1] 1788528899.154343 mapwrite 0x8c2053 thru 0x8c9b58 (0x7b06 bytes) 000185[1] 1788528899.393559 trunc from 0x8c9b59 to 0x5789c3 (0x351195 bytes) 000186[0] 1788528899.468543 mapread 0x2c7454 thru 0x2c76e9 (0x296 bytes) 000188[0] 1788528899.539716 read 0x29fe1f thru 0x2ae7cb (0xe9ad bytes) 000189[1] 1788528899.693000 trunc from 0x5789c4 to 0x341e22 (0x236ba1 bytes) 000191[1] 1788528899.802967 write 0x8d8087 thru 0x8dcf48 (0x4ec2 bytes) 000192[1] 1788528899.809252 read 0x0bcc8b thru 0x0bd1c2 (0x538 bytes) 000193[1] 1788528899.824556 mapread 0x268784 thru 0x26f829 (0x70a6 bytes) 000194[1] 1788528899.878136 read 0x8da9f0 thru 0x8db2cf (0x8e0 bytes) 000197[0] 1788528899.988350 read 0x62daa5 thru 0x62f449 (0x19a5 bytes) 000198[0] 1788528900.065738 trunc from 0x8dcf49 to 0x3d0753 (0x50c7f5 bytes) 000199[1] 1788528900.158940 mapread 0x1c8bc0 thru 0x1d4a72 (0xbeb3 bytes) 000200[0] 1788528900.302702 punch from 0x3b4160 to 0x3c24fd (0xe39e bytes) 000203[1] 1788528900.400534 write 0x376bc8 thru 0x384c1c (0xe055 bytes) 000205[1] 1788528900.517536 read 0x10cbaa thru 0x11b58a (0xe9e1 bytes) 000207[0] 1788528900.663955 punch from 0x12f4f2 to 0x1312f7 (0x1e06 bytes) 000210[1] 1788528900.785389 punch from 0x055226 to 0x05db2f (0x890a bytes) 000212[0] 1788528900.873879 mapread 0x121a30 thru 0x12c113 (0xa6e4 bytes) 000216[0] 1788528900.994210 mapwrite 0x7621f4 thru 0x769bd4 (0x79e1 bytes) 000218[0] 1788528901.150622 read 0x11db7d thru 0x1205c7 (0x2a4b bytes) 000219[1] 1788528901.158832 trunc from 0x769bd5 to 0x4d44ed (0x2956e7 bytes) 000220[1] 1788528901.212161 write 0x3b65ad thru 0x3ba75f (0x41b3 bytes) 000221[0] 1788528901.251593 trunc from 0x4d44ee to 0x45ed08 (0x757e5 bytes) 000226[0] 1788528901.399099 trunc from 0x45ed09 to 0x4093ac (0x5595c bytes) 000228[1] 1788528901.426138 mapread 0x236952 thru 0x241bfc (0xb2ab bytes) 000229[0] 1788528901.542294 trunc from 0x4093ad to 0x30c176 (0xfd236 bytes) 000231[1] 1788528901.589134 mapread 0x1bdd2d thru 0x1cbf2b (0xe1ff bytes) 000234[0] 1788528901.702925 mapread 0x29e47b thru 0x2ab521 (0xd0a7 bytes) 000240[1] 1788528901.773103 mapread 0x16a7b8 thru 0x176f1c (0xc765 bytes) 000242[0] 1788528901.890778 mapwrite 0x8f3d3e thru 0x8f579f (0x1a62 bytes) 000243[0] 1788528902.026850 punch from 0x1cb4d4 to 0x1d1ba5 (0x66d2 bytes) 000244[1] 1788528902.036649 read 0x24b4cc thru 0x25711b (0xbc50 bytes) 000248[0] 1788528902.091602 trunc from 0x8f57a0 to 0x7e3a69 (0x111d36 bytes) 000250[1] 1788528902.183905 mapread 0x0e64bd thru 0x0ec639 (0x617d bytes) 000257[0] 1788528902.275791 read 0x3e45fd thru 0x3ed8c4 (0x92c8 bytes) 000258[0] 1788528902.287466 trunc from 0x7e3a6a to 0x019b85 (0x7c9ee4 bytes) 000260[0] 1788528902.353221 mapwrite 0x56bf81 thru 0x57ba23 (0xfaa3 bytes) 000262[0] 1788528902.463877 trunc from 0x57ba24 to 0x541f36 (0x39aed bytes) 000265[0] 1788528902.540834 punch from 0x225d04 to 0x22f87f (0x9b7c bytes) 000266[0] 1788528902.545100 mapwrite 0x0a3dcd thru 0x0abe50 (0x8084 bytes) 000268[1] 1788528902.661353 punch from 0x43e29e to 0x44bb69 (0xd8cc bytes) 000271[0] 1788528902.721467 trunc from 0x541f37 to 0x7372d2 (0x1f539c bytes) 000273[0] 1788528902.784551 read 0x5a272d thru 0x5ad9b8 (0xb28c bytes) 000274[0] 1788528902.897439 write 0x669a4d thru 0x673e2b (0xa3df bytes) 000275[0] 1788528903.018312 mapread 0x5ee4a6 thru 0x5f3f1a (0x5a75 bytes) 000276[1] 1788528903.081545 read 0x31f18c thru 0x32a50a (0xb37f bytes) 000278[0] 1788528903.153343 write 0x3b2dc1 thru 0x3b4322 (0x1562 bytes) 000279[1] 1788528903.191014 mapread 0x680b92 thru 0x68bc52 (0xb0c1 bytes) 000280[0] 1788528903.283044 read 0x2edf4d thru 0x2f176d (0x3821 bytes) 000281[1] 1788528903.307125 write 0x5b186c thru 0x5b67a1 (0x4f36 bytes) 000285[0] 1788528903.477952 mapread 0x047758 thru 0x04f89b (0x8144 bytes) 000286[1] 1788528903.582560 write 0x043323 thru 0x05033f (0xd01d bytes) 000287[1] 1788528903.732407 read 0x495102 thru 0x4a4f06 (0xfe05 bytes) 000290[1] 1788528903.845733 mapread 0x5c9b77 thru 0x5c9ed0 (0x35a bytes) 000291[1] 1788528903.900247 write 0x9f4982 thru 0x9fe193 (0x9812 bytes) 000294[0] 1788528903.972763 read 0x29c3e2 thru 0x29dca6 (0x18c5 bytes) 000295[0] 1788528904.033338 mapread 0x05b06c thru 0x066d0f (0xbca4 bytes) 000297[1] 1788528904.132350 punch from 0x5910bc to 0x59cfa3 (0xbee8 bytes) 000298[0] 1788528904.172024 mapwrite 0x70cc2d thru 0x71beeb (0xf2bf bytes) 000299[0] 1788528904.351202 trunc from 0x9fe194 to 0x6198df (0x3e48b4 bytes) 000301[0] 1788528904.403124 trunc from 0x6198e0 to 0x8f77a2 (0x2ddec3 bytes) 000304[0] 1788528904.491374 read 0x18040e thru 0x181179 (0xd6c bytes) 000305[0] 1788528904.526584 read 0x6f59db thru 0x7001b8 (0xa7de bytes) 000310[0] 1788528904.651616 punch from 0x70a3e8 to 0x710d36 (0x694f bytes) 000311[1] 1788528904.659791 trunc from 0x8f77a3 to 0x8d5b37 (0x21c6b bytes) 000312[0] 1788528904.716514 trunc from 0x8d5b38 to 0x6c9e74 (0x20bcc3 bytes) 000315[0] 1788528904.800901 trunc from 0x6c9e75 to 0x17b80d (0x54e667 bytes) 000318[0] 1788528904.900211 punch from 0x0c5575 to 0x0c8b58 (0x35e4 bytes) 000319[1] 1788528904.908727 mapread 0x11773c thru 0x120fa0 (0x9865 bytes) 000320[0] 1788528905.007373 read 0x09da2f thru 0x09db93 (0x165 bytes) 000321[1] 1788528905.059906 read 0x029f9f thru 0x02d6fd (0x375f bytes) 000322[1] 1788528905.149598 mapwrite 0x2d4cbd thru 0x2daf6f (0x62b3 bytes) 000325[0] 1788528905.599147 punch from 0x0b206b to 0x0ba2f6 (0x828c bytes) 000326[1] 1788528905.667815 mapread 0x241865 thru 0x24af44 (0x96e0 bytes) 000327[0] 1788528905.805833 mapread 0x16cf08 thru 0x17c35e (0xf457 bytes) 000328[0] 1788528905.941533 mapwrite 0x78955d thru 0x790c27 (0x76cb bytes) 000329[0] 1788528906.192683 mapwrite 0x760a78 thru 0x766961 (0x5eea bytes) 000332[1] 1788528906.298151 read 0x4ac27e thru 0x4ae8b2 (0x2635 bytes) 000333[1] 1788528906.359375 punch from 0x5de386 to 0x5e15b7 (0x3232 bytes) 000334[0] 1788528906.419149 read 0x6dcc7d thru 0x6dff04 (0x3288 bytes) 000337[0] 1788528906.504838 punch from 0x26f92a to 0x27e98e (0xf065 bytes) 000339[0] 1788528906.562244 trunc from 0x790c28 to 0x66fe0a (0x120e1d bytes) 000340[0] 1788528906.604666 mapwrite 0x7ca3e5 thru 0x7d647c (0xc098 bytes) 000341[0] 1788528906.713796 read 0x4167e4 thru 0x4177b7 (0xfd4 bytes) 000343[1] 1788528906.782524 punch from 0x0c4ea6 to 0x0d36a8 (0xe803 bytes) 000344[1] 1788528906.819251 trunc from 0x7d647d to 0x700219 (0xd6263 bytes) 000345[0] 1788528906.859607 mapwrite 0x8e0fd9 thru 0x8e4f38 (0x3f60 bytes) 000349[1] 1788528907.109529 punch from 0x2689eb to 0x27370e (0xad24 bytes) 000351[1] 1788528907.194150 mapwrite 0x13cd34 thru 0x14ae31 (0xe0fe bytes) 000353[1] 1788528907.390655 read 0x3f06c3 thru 0x3f2064 (0x19a2 bytes) 000354[0] 1788528907.430439 read 0x0402c0 thru 0x04a668 (0xa3a9 bytes) 000355[1] 1788528907.477786 read 0x801917 thru 0x8115f5 (0xfcdf bytes) 000357[1] 1788528907.525905 punch from 0x5a8475 to 0x5b765c (0xf1e8 bytes) 000358[0] 1788528907.561719 read 0x3f3388 thru 0x3f6eb4 (0x3b2d bytes) 000360[0] 1788528907.584814 mapread 0x548b45 thru 0x555feb (0xd4a7 bytes) 000361[0] 1788528907.682170 mapwrite 0x37e511 thru 0x384931 (0x6421 bytes) 000362[1] 1788528907.799849 read 0x3d6a09 thru 0x3dfb9a (0x9192 bytes) 000363[1] 1788528907.824015 mapread 0x888285 thru 0x895632 (0xd3ae bytes) 000366[0] 1788528907.854583 mapwrite 0x4ffbd9 thru 0x500910 (0xd38 bytes) 000367[0] 1788528907.951882 trunc from 0x8e4f39 to 0x72d9ed (0x1b754b bytes) 000369[0] 1788528908.101655 punch from 0x6c324b to 0x6ce16d (0xaf23 bytes) 000370[0] 1788528908.152081 punch from 0x10b872 to 0x112bf0 (0x737f bytes) 000371[1] 1788528908.180564 write 0x403bff thru 0x40aaf3 (0x6ef5 bytes) 000372[0] 1788528908.215379 mapwrite 0x7efb45 thru 0x7f9b98 (0xa054 bytes) 000373[0] 1788528908.340895 mapread 0x330ce7 thru 0x332bf8 (0x1f12 bytes) 000374[1] 1788528908.358951 trunc from 0x7f9b99 to 0x263cdb (0x595ebd bytes) 000379[1] 1788528908.450735 write 0x6c21dd thru 0x6c3556 (0x137a bytes) 000380[1] 1788528908.455594 punch from 0x352274 to 0x35b280 (0x900d bytes) 000381[0] 1788528908.466424 mapread 0x0fe302 thru 0x10c2ab (0xdfaa bytes) 000384[0] 1788528908.622816 mapwrite 0x12d28d thru 0x12fe8e (0x2c02 bytes) 000385[0] 1788528908.765627 mapwrite 0x72031e thru 0x72c9e1 (0xc6c4 bytes) 000387[0] 1788528908.997068 mapread 0x48f22e thru 0x49a06f (0xae42 bytes) 000388[1] 1788528909.041441 mapread 0x25f7e8 thru 0x26e0b1 (0xe8ca bytes) truncating to largest ever: 0x99a7f9 000389[0] 1788528909.131487 trunc from 0x72c9e2 to 0x99a7f8 (0x26de17 bytes) 000392[1] 1788528909.277477 mapread 0x515caf thru 0x51ed78 (0x90ca bytes) 000393[0] 1788528909.341356 write 0x5e3f78 thru 0x5ec749 (0x87d2 bytes) 000394[0] 1788528909.414025 read 0x2ba34a thru 0x2c2f1a (0x8bd1 bytes) 000395[0] 1788528909.471936 read 0x203533 thru 0x20e958 (0xb426 bytes) 000396[1] 1788528909.488942 mapwrite 0x2cc6c8 thru 0x2d7261 (0xab9a bytes) 000398[1] 1788528909.780148 mapread 0x6c50cb thru 0x6cf57d (0xa4b3 bytes) 000400[1] 1788528909.889059 mapwrite 0x4cb659 thru 0x4cd1ba (0x1b62 bytes) 000401[0] 1788528910.036126 mapread 0x678a0b thru 0x688885 (0xfe7b bytes) 000402[0] 1788528910.170022 mapwrite 0x1ddbab thru 0x1e2a29 (0x4e7f bytes) 000404[0] 1788528910.410980 trunc from 0x99a7f9 to 0x858879 (0x141f7f bytes) 000405[1] 1788528910.458190 read 0x2f668b thru 0x2ff4db (0x8e51 bytes) 000407[1] 1788528910.513168 punch from 0x0c4192 to 0x0cac98 (0x6b07 bytes) 000408[1] 1788528910.545608 mapread 0x66b4e4 thru 0x6720bc (0x6bd9 bytes) 000409[1] 1788528910.609037 write 0x5c90b2 thru 0x5cb3a2 (0x22f1 bytes) 000410[0] 1788528910.664664 write 0x9d7524 thru 0x9e42fb (0xcdd8 bytes) 000413[0] 1788528910.731932 read 0x991753 thru 0x99db27 (0xc3d5 bytes) 000414[1] 1788528910.753999 punch from 0x37480a to 0x381620 (0xce17 bytes) 000415[1] 1788528910.801318 trunc from 0x9e42fc to 0x5e3a7a (0x400881 bytes) 000416[1] 1788528910.893353 mapread 0x32c5c9 thru 0x32e4f2 (0x1f2a bytes) 000417[1] 1788528910.945807 mapwrite 0x1c3a84 thru 0x1caea1 (0x741e bytes) 000420[1] 1788528911.033099 trunc from 0x5e3a7b to 0x564865 (0x7f215 bytes) 000421[1] 1788528911.094308 read 0x02b309 thru 0x02c064 (0xd5c bytes) 000423[1] 1788528911.174080 punch from 0x17b17f to 0x189357 (0xe1d9 bytes) 000424[1] 1788528911.189379 punch from 0x2f0733 to 0x2f3b7d (0x344b bytes) 000427[0] 1788528911.287023 mapread 0x3397d8 thru 0x34792a (0xe153 bytes) 000430[1] 1788528911.366162 read 0x47d137 thru 0x4896be (0xc588 bytes) 000432[0] 1788528911.400175 punch from 0x34e7ed to 0x350ffa (0x280e bytes) 000433[1] 1788528911.446455 mapread 0x29aa13 thru 0x2a81a9 (0xd797 bytes) 000434[1] 1788528911.515224 read 0x4d248e thru 0x4d9bd1 (0x7744 bytes) 000436[1] 1788528911.561600 mapread 0x37f7cc thru 0x381468 (0x1c9d bytes) 000438[0] 1788528911.585077 write 0x8ad762 thru 0x8af524 (0x1dc3 bytes) 000440[0] 1788528911.644394 mapread 0x3041db thru 0x30ce16 (0x8c3c bytes) 000443[1] 1788528911.793136 write 0x372398 thru 0x376298 (0x3f01 bytes) 000444[1] 1788528911.861668 mapwrite 0x47e375 thru 0x4822ba (0x3f46 bytes) 000446[1] 1788528911.979233 trunc from 0x8af525 to 0x7584a2 (0x157082 bytes) 000447[0] 1788528912.007251 mapread 0x35d59d thru 0x367245 (0x9ca9 bytes) 000448[0] 1788528912.120146 punch from 0x467014 to 0x4747c2 (0xd7af bytes) 000449[1] 1788528912.164948 punch from 0x17cdb5 to 0x17e065 (0x12b1 bytes) 000450[1] 1788528912.207437 trunc from 0x7584a3 to 0x8aee52 (0x1569b0 bytes) 000452[1] 1788528912.294496 mapwrite 0x0123b9 thru 0x017e79 (0x5ac1 bytes) 000453[0] 1788528912.398615 trunc from 0x8aee53 to 0x93d2b6 (0x8e464 bytes) 000455[0] 1788528912.527962 punch from 0x771e06 to 0x771fb5 (0x1b0 bytes) 000456[1] 1788528912.544341 write 0x6f0d0c thru 0x6f182f (0xb24 bytes) 000458[1] 1788528912.604345 punch from 0x385493 to 0x387688 (0x21f6 bytes) 000459[1] 1788528912.649364 mapread 0x286bf2 thru 0x2902c9 (0x96d8 bytes) 000462[1] 1788528912.710966 write 0x4ba54e thru 0x4c4cd7 (0xa78a bytes) 000463[0] 1788528912.782714 read 0x1b6567 thru 0x1bb888 (0x5322 bytes) truncating to largest ever: 0x9d26e2 000465[0] 1788528912.889794 trunc from 0x93d2b7 to 0x9d26e1 (0x9542b bytes) 000466[1] 1788528913.029505 write 0x584247 thru 0x58738b (0x3145 bytes) 000467[1] 1788528913.071559 read 0x714355 thru 0x71c403 (0x80af bytes) 000469[0] 1788528913.114141 mapread 0x23e762 thru 0x23eaa5 (0x344 bytes) 000470[0] 1788528913.144207 write 0x28b60f thru 0x298fb3 (0xd9a5 bytes) 000471[1] 1788528913.174492 mapwrite 0x3b1b17 thru 0x3b589e (0x3d88 bytes) 000472[0] 1788528913.353512 punch from 0x86c588 to 0x86d3a2 (0xe1b bytes) 000473[1] 1788528913.403930 mapwrite 0x656530 thru 0x65db15 (0x75e6 bytes) 000475[1] 1788528913.559729 mapwrite 0x0fc853 thru 0x1086c0 (0xbe6e bytes) 000476[0] 1788528913.702986 write 0x37ec7c thru 0x384ba1 (0x5f26 bytes) 000477[1] 1788528913.750692 punch from 0x0e292c to 0x0eb30e (0x89e3 bytes) 000481[0] 1788528913.822436 mapwrite 0x793669 thru 0x797760 (0x40f8 bytes) 000483[1] 1788528914.023015 read 0x8f903a thru 0x8fe818 (0x57df bytes) 000484[1] 1788528914.053488 read 0x5c84d2 thru 0x5d0a5f (0x858e bytes) 000485[0] 1788528914.105334 write 0x117e0a thru 0x11b679 (0x3870 bytes) 000486[1] 1788528914.147392 mapread 0x6b1188 thru 0x6bf7b6 (0xe62f bytes) 000489[1] 1788528914.225558 write 0x26f88b thru 0x27ddb7 (0xe52d bytes) 000490[0] 1788528914.271444 mapwrite 0x1cc601 thru 0x1d3162 (0x6b62 bytes) 000491[0] 1788528914.431357 mapread 0x102ef7 thru 0x103081 (0x18b bytes) 000493[0] 1788528914.487126 mapread 0x0bee60 thru 0x0bf009 (0x1aa bytes) 000494[0] 1788528914.520179 mapread 0x2e624d thru 0x2f11ba (0xaf6e bytes) 000496[0] 1788528914.593453 punch from 0x806089 to 0x81272f (0xc6a7 bytes) 000497[1] 1788528914.616666 trunc from 0x9d26e2 to 0x86fbf6 (0x162aeb bytes) 000498[1] 1788528914.701002 read 0x014961 thru 0x02361a (0xecba bytes) 000499[0] 1788528914.741464 write 0x4ed38b thru 0x4f99eb (0xc661 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8914 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero length punch hole 000003[0] 1788528915.468348 write 0x08e000 thru 0x09b950 (0xd951 bytes) 000004[0] 1788528915.492194 write 0x2f7000 thru 0x2fbec1 (0x4ec2 bytes) truncating to largest ever: 0x59e811 000007[1] 1788528915.517578 trunc from 0x2fbec2 to 0x59e810 (0x2a294f bytes) truncating to largest ever: 0x736c10 000009[0] 1788528915.609510 trunc from 0x59e811 to 0x736c0f (0x1983ff bytes) 000010[0] 1788528915.696196 trunc from 0x736c10 to 0x1d73a1 (0x55f86e bytes) 000011[0] 1788528915.769174 trunc from 0x1d73a2 to 0x5d3a03 (0x3fc662 bytes) 000012[1] 1788528915.849330 read_OD 0x29e000 thru 0x2a5fff (0x8000 bytes) 000015[0] 1788528915.902162 punch from 0x52740d to 0x527666 (0x25a bytes) 000018[0] 1788528916.013493 read 0x350000 thru 0x35e48d (0xe48e bytes) 000025[0] 1788528916.132192 write 0x739000 thru 0x73a44d (0x144e bytes) 000026[1] 1788528916.135042 mapread 0x30f000 thru 0x315eb6 (0x6eb7 bytes) 000027[0] 1788528916.212470 write 0x165000 thru 0x1658e7 (0x8e8 bytes) 000028[1] 1788528916.247937 mapread 0x3c9000 thru 0x3d21cb (0x91cc bytes) 000029[0] 1788528916.331055 punch from 0x71610e to 0x71a22c (0x411f bytes) 000030[1] 1788528916.406661 write_OD 0x2d2000 thru 0x2d6fff (0x5000 bytes) 000031[0] 1788528916.522959 read 0x732000 thru 0x738e9e (0x6e9f bytes) 000032[1] 1788528916.545298 punch from 0x505a34 to 0x513e67 (0xe434 bytes) 000033[0] 1788528916.585334 punch from 0x05c88b to 0x06856c (0xbce2 bytes) 000035[1] 1788528916.609163 read_OD 0x5fc000 thru 0x606fff (0xb000 bytes) 000036[1] 1788528916.638476 punch from 0x4b7050 to 0x4bbccb (0x4c7c bytes) 000037[0] 1788528916.654125 trunc from 0x73a44e to 0x0e0836 (0x659c17 bytes) 000039[0] 1788528916.748355 mapread 0x099000 thru 0x0a5319 (0xc31a bytes) 000041[0] 1788528916.870617 mapwrite 0x43a000 thru 0x43cc6a (0x2c6b bytes) 000042[1] 1788528916.993354 mapread 0x0e7000 thru 0x0e7cd5 (0xcd6 bytes) 000043[0] 1788528917.031146 mapwrite 0x42f000 thru 0x4335c5 (0x45c6 bytes) truncating to largest ever: 0x883474 000046[0] 1788528917.150645 trunc from 0x43cc6b to 0x883473 (0x446809 bytes) 000048[1] 1788528917.253056 mapread 0x155000 thru 0x155b5b (0xb5c bytes) 000051[1] 1788528917.295969 trunc from 0x883474 to 0x13ecca (0x7447a9 bytes) truncating to largest ever: 0x900d1c 000052[0] 1788528917.350029 trunc from 0x13eccb to 0x900d1b (0x7c2051 bytes) 000054[1] 1788528917.409998 mapwrite 0x92f000 thru 0x935a55 (0x6a56 bytes) 000055[0] 1788528917.629041 punch from 0x801d56 to 0x8030ef (0x139a bytes) 000056[1] 1788528917.690253 mapwrite 0x33d000 thru 0x347182 (0xa183 bytes) 000057[0] 1788528917.888412 punch from 0x756ea3 to 0x766783 (0xf8e1 bytes) 000060[0] 1788528917.957500 trunc from 0x935a56 to 0x76321a (0x1d283b bytes) 000061[1] 1788528918.012419 write_OD 0x7c1000 thru 0x7c2fff (0x2000 bytes) 000062[1] 1788528918.070700 punch from 0x4bf5c2 to 0x4c39e7 (0x4426 bytes) 000063[0] 1788528918.088400 write 0x958000 thru 0x965c52 (0xdc53 bytes) 000065[1] 1788528918.104377 read_OD 0x5a1000 thru 0x5abfff (0xb000 bytes) 000067[1] 1788528918.145176 read_OD 0x5ac000 thru 0x5b9fff (0xe000 bytes) 000069[1] 1788528918.154106 mapwrite 0x977000 thru 0x979025 (0x2026 bytes) 000070[0] 1788528918.298519 mapwrite 0x9a8000 thru 0x9b2fce (0xafcf bytes) 000072[0] 1788528918.497564 write 0x78b000 thru 0x78f517 (0x4518 bytes) 000074[1] 1788528918.550241 punch from 0x672864 to 0x679d0b (0x74a8 bytes) 000075[0] 1788528918.602972 write 0x989000 thru 0x98f757 (0x6758 bytes) 000077[1] 1788528918.633137 punch from 0x630d30 to 0x63b3f1 (0xa6c2 bytes) 000078[0] 1788528918.674229 mapwrite 0x1f7000 thru 0x1fac6f (0x3c70 bytes) 000080[1] 1788528918.834054 mapread 0x87f000 thru 0x88eece (0xfecf bytes) 000082[1] 1788528918.981426 read_OD 0x824000 thru 0x82bfff (0x8000 bytes) truncating to largest ever: 0x9b7be0 000084[1] 1788528918.998237 trunc from 0x9b2fcf to 0x9b7bdf (0x4c11 bytes) 000085[1] 1788528919.069305 write_OD 0x0f1000 thru 0x0f2fff (0x2000 bytes) 000086[0] 1788528919.207515 write 0x602000 thru 0x60a76d (0x876e bytes) 000087[1] 1788528919.276870 mapread 0x05c000 thru 0x05cf6c (0xf6d bytes) 000088[0] 1788528919.335085 read 0x803000 thru 0x80bf80 (0x8f81 bytes) 000090[0] 1788528919.348123 trunc from 0x9b7be0 to 0x54caa5 (0x46b13a bytes) 000092[1] 1788528919.427851 mapwrite 0x32d000 thru 0x33b9be (0xe9bf bytes) 000093[0] 1788528919.568794 write 0x61b000 thru 0x622cb4 (0x7cb5 bytes) 000095[1] 1788528919.615750 write_OD 0x2d1000 thru 0x2ddfff (0xd000 bytes) 000096[1] 1788528919.761140 mapwrite 0x2fa000 thru 0x307f69 (0xdf6a bytes) 000097[0] 1788528919.905413 read 0x210000 thru 0x21cd6f (0xcd70 bytes) 000098[1] 1788528919.939468 write_OD 0x3e8000 thru 0x3f0fff (0x9000 bytes) 000099[1] 1788528920.011487 read_OD 0x211000 thru 0x213fff (0x3000 bytes) 000100[1] 1788528920.022025 trunc from 0x622cb5 to 0x61c767 (0x654d bytes) 000101[0] 1788528920.062431 read 0x0fe000 thru 0x0ff6bb (0x16bc bytes) 000103[1] 1788528920.090942 mapwrite 0x0ab000 thru 0x0ad99b (0x299c bytes) 000105[1] 1788528920.201240 punch from 0x56813b to 0x57040a (0x82d0 bytes) 000106[1] 1788528920.227855 mapread 0x3ec000 thru 0x3f6642 (0xa643 bytes) 000107[1] 1788528920.250233 mapread 0x0f0000 thru 0x0ffd53 (0xfd54 bytes) truncating to largest ever: 0x9db2a2 000110[0] 1788528920.336219 trunc from 0x61c768 to 0x9db2a1 (0x3beb3a bytes) 000111[1] 1788528920.433350 mapwrite 0x567000 thru 0x56e447 (0x7448 bytes) 000112[1] 1788528920.638189 trunc from 0x9db2a2 to 0x808d1c (0x1d2585 bytes) 000113[0] 1788528920.678462 mapwrite 0x4a8000 thru 0x4aacac (0x2cad bytes) 000114[0] 1788528920.955976 trunc from 0x808d1d to 0x82360c (0x1a8f0 bytes) 000116[1] 1788528921.042735 trunc from 0x82360d to 0x850e0c (0x2d800 bytes) 000117[1] 1788528921.161035 mapread 0x47d000 thru 0x48269b (0x569c bytes) 000118[0] 1788528921.226469 mapwrite 0x35d000 thru 0x3647f8 (0x77f9 bytes) 000120[1] 1788528921.483165 punch from 0x3ab04f to 0x3ba4e6 (0xf498 bytes) 000122[0] 1788528921.545186 trunc from 0x850e0d to 0x1b8c84 (0x698188 bytes) 000123[0] 1788528921.583391 punch from 0x0eb770 to 0x0f41ba (0x8a4b bytes) 000129[0] 1788528921.642475 mapread 0x178000 thru 0x181fa2 (0x9fa3 bytes) 000130[1] 1788528921.672938 read_OD 0x062000 thru 0x067fff (0x6000 bytes) 000134[1] 1788528921.704981 trunc from 0x1b8c85 to 0x5fcca5 (0x444021 bytes) 000135[0] 1788528921.740887 write 0x03a000 thru 0x049c67 (0xfc68 bytes) 000140[0] 1788528921.825996 read 0x479000 thru 0x479858 (0x859 bytes) 000143[1] 1788528921.868214 punch from 0x347830 to 0x34a839 (0x300a bytes) 000144[0] 1788528921.900939 mapread 0x5a3000 thru 0x5b1cf7 (0xecf8 bytes) 000145[0] 1788528921.958343 punch from 0x129ba3 to 0x130568 (0x69c6 bytes) 000147[0] 1788528922.006029 mapread 0x32d000 thru 0x3371dd (0xa1de bytes) 000148[1] 1788528922.042626 trunc from 0x5fcca6 to 0x64e2e6 (0x51641 bytes) 000149[0] 1788528922.147999 punch from 0x20c935 to 0x213d8c (0x7458 bytes) 000151[0] 1788528922.197438 read 0x228000 thru 0x2348a9 (0xc8aa bytes) 000153[0] 1788528922.233798 write 0x838000 thru 0x83cf79 (0x4f7a bytes) 000154[1] 1788528922.237396 punch from 0x0a7f37 to 0x0a8126 (0x1f0 bytes) 000158[1] 1788528922.320788 trunc from 0x83cf7a to 0x619c8f (0x2232ea bytes) 000162[1] 1788528922.402034 mapread 0x2fa000 thru 0x2fa078 (0x79 bytes) 000165[1] 1788528922.506602 mapwrite 0x6c5000 thru 0x6d204d (0xd04e bytes) 000166[0] 1788528922.668367 mapwrite 0x27b000 thru 0x287a83 (0xca84 bytes) 000167[0] 1788528922.771607 read 0x0d8000 thru 0x0dcfcd (0x4fce bytes) 000168[1] 1788528922.780384 write_OD 0x20c000 thru 0x211fff (0x6000 bytes) 000169[1] 1788528922.838437 punch from 0x4b1851 to 0x4bfea6 (0xe656 bytes) 000170[1] 1788528922.868104 write_OD 0x062000 thru 0x06efff (0xd000 bytes) 000171[1] 1788528922.933458 mapwrite 0x28c000 thru 0x298874 (0xc875 bytes) 000172[0] 1788528923.088621 read 0x0a9000 thru 0x0ab928 (0x2929 bytes) 000180[0] 1788528923.116531 write 0x79f000 thru 0x7aceb3 (0xdeb4 bytes) 000182[0] 1788528923.133802 write 0x7cb000 thru 0x7cb136 (0x137 bytes) 000184[1] 1788528923.168878 punch from 0x1703ee to 0x1756dd (0x52f0 bytes) 000185[1] 1788528923.194708 punch from 0x78b6f9 to 0x791170 (0x5a78 bytes) 000186[1] 1788528923.199033 trunc from 0x7cb137 to 0x3e0efc (0x3ea23a bytes) 000187[1] 1788528923.236342 punch from 0x170dbe to 0x172c53 (0x1e96 bytes) 000188[1] 1788528923.244199 read_OD 0x041000 thru 0x043fff (0x3000 bytes) 000189[0] 1788528923.258837 read 0x085000 thru 0x089fba (0x4fbb bytes) 000191[0] 1788528923.328979 read 0x37a000 thru 0x385a27 (0xba28 bytes) 000193[1] 1788528923.329673 read_OD 0x043000 thru 0x04ffff (0xd000 bytes) 000194[0] 1788528923.342179 read 0x203000 thru 0x212948 (0xf949 bytes) 000196[1] 1788528923.343160 trunc from 0x3e0efd to 0x521f3d (0x141041 bytes) 000197[0] 1788528923.403251 write 0x35d000 thru 0x363ec8 (0x6ec9 bytes) 000198[1] 1788528923.494767 read_OD 0x4bd000 thru 0x4c0fff (0x4000 bytes) 000201[1] 1788528923.548869 mapwrite 0x884000 thru 0x88a477 (0x6478 bytes) 000203[1] 1788528923.672057 write_OD 0x61c000 thru 0x61ffff (0x4000 bytes) 000204[1] 1788528923.723191 read_OD 0x793000 thru 0x7a0fff (0xe000 bytes) 000205[0] 1788528923.731584 write 0x2e5000 thru 0x2e742e (0x242f bytes) 000206[1] 1788528923.767678 read_OD 0x666000 thru 0x66efff (0x9000 bytes) 000211[1] 1788528923.809706 read_OD 0x138000 thru 0x140fff (0x9000 bytes) 000214[0] 1788528923.818846 trunc from 0x88a478 to 0x20640c (0x68406b bytes) 000215[0] 1788528923.870994 punch from 0x175cea to 0x17a2e6 (0x45fd bytes) 000216[0] 1788528923.888369 mapwrite 0x5cd000 thru 0x5cfda8 (0x2da9 bytes) 000220[0] 1788528924.035436 mapwrite 0x8f0000 thru 0x8fb46f (0xb470 bytes) 000221[1] 1788528924.224003 mapwrite 0x3eb000 thru 0x3ef2b5 (0x42b6 bytes) 000222[0] 1788528924.417341 mapwrite 0x60e000 thru 0x6116b3 (0x36b4 bytes) 000225[0] 1788528924.727319 write 0x785000 thru 0x794acc (0xfacd bytes) 000227[1] 1788528924.777826 read_OD 0x34c000 thru 0x353fff (0x8000 bytes) 000228[1] 1788528924.875912 read_OD 0x69f000 thru 0x6acfff (0xe000 bytes) 000229[0] 1788528924.892655 read 0x83c000 thru 0x83e913 (0x2914 bytes) 000230[1] 1788528924.920197 trunc from 0x8fb470 to 0x964296 (0x68e27 bytes) 000231[0] 1788528924.999341 read 0x0cb000 thru 0x0d0695 (0x5696 bytes) 000232[1] 1788528925.030506 mapread 0x6f2000 thru 0x701f38 (0xff39 bytes) 000233[1] 1788528925.066163 write_OD 0x55c000 thru 0x55cfff (0x1000 bytes) 000234[1] 1788528925.137395 read_OD 0x377000 thru 0x381fff (0xb000 bytes) 000236[0] 1788528925.163746 mapwrite 0x8f6000 thru 0x8f6080 (0x81 bytes) 000238[1] 1788528925.284963 trunc from 0x964297 to 0x587e37 (0x3dc45f bytes) 000240[1] 1788528925.499906 mapwrite 0x949000 thru 0x952f45 (0x9f46 bytes) 000241[0] 1788528925.719340 trunc from 0x952f46 to 0x2cdc70 (0x6852d5 bytes) 000242[1] 1788528925.770987 punch from 0x2bb2a5 to 0x2c9d68 (0xeac4 bytes) 000243[0] 1788528925.798921 read 0x1ef000 thru 0x1ef6bc (0x6bd bytes) 000244[0] 1788528925.815444 trunc from 0x2cdc71 to 0x3d0752 (0x102ae2 bytes) 000249[1] 1788528925.960464 mapwrite 0x9ea000 thru 0x9f9337 (0xf338 bytes) 000250[0] 1788528926.083103 mapread 0x660000 thru 0x66f4d6 (0xf4d7 bytes) 000251[0] 1788528926.161566 read 0x77d000 thru 0x782008 (0x5009 bytes) 000252[0] 1788528926.178021 read 0x219000 thru 0x22359d (0xa59e bytes) 000253[1] 1788528926.194756 read_OD 0x6c9000 thru 0x6cafff (0x2000 bytes) 000254[0] 1788528926.203132 mapread 0x9c9000 thru 0x9cea24 (0x5a25 bytes) 000257[0] 1788528926.259441 read 0x7a9000 thru 0x7b6af9 (0xdafa bytes) 000264[0] 1788528926.278930 trunc from 0x9f9338 to 0x139b89 (0x8bf7ae bytes) 000266[1] 1788528926.355802 mapwrite 0x8f8000 thru 0x8fd0a3 (0x50a4 bytes) 000268[1] 1788528926.496444 trunc from 0x8fd0a4 to 0x967a2f (0x6a98c bytes) 000269[1] 1788528926.517181 mapread 0x434000 thru 0x4353d8 (0x13d9 bytes) 000272[1] 1788528926.564274 punch from 0x81c928 to 0x82b4be (0xeb97 bytes) 000275[0] 1788528926.635326 mapread 0x243000 thru 0x245241 (0x2242 bytes) 000278[0] 1788528926.726227 write 0x85e000 thru 0x86b4c9 (0xd4ca bytes) 000279[0] 1788528926.835510 write 0x4fc000 thru 0x506189 (0xa18a bytes) 000280[0] 1788528926.853114 read 0x686000 thru 0x69155a (0xb55b bytes) 000281[1] 1788528926.880337 read 0x802000 thru 0x809681 (0x7682 bytes) 000283[1] 1788528926.944853 trunc from 0x967a30 to 0x728c1e (0x23ee11 bytes) 000285[0] 1788528926.991110 read 0x1db000 thru 0x1e1df0 (0x6df1 bytes) 000286[0] 1788528927.017656 mapwrite 0x9ed000 thru 0x9f9419 (0xc41a bytes) 000288[0] 1788528927.155886 mapwrite 0x5ab000 thru 0x5ae0e1 (0x30e2 bytes) 000289[1] 1788528927.239237 mapread 0x21c000 thru 0x22bde8 (0xfde9 bytes) 000291[0] 1788528927.297432 write 0x4d4000 thru 0x4d5681 (0x1682 bytes) 000294[0] 1788528927.348960 punch from 0x363472 to 0x366478 (0x3007 bytes) 000295[1] 1788528927.374689 punch from 0x63d236 to 0x642efc (0x5cc7 bytes) 000297[1] 1788528927.408981 read 0x0dd000 thru 0x0e2fd9 (0x5fda bytes) 000298[0] 1788528927.454002 write 0x6b1000 thru 0x6b16f3 (0x6f4 bytes) 000300[0] 1788528927.565008 punch from 0x6779e6 to 0x683651 (0xbc6c bytes) 000301[0] 1788528927.587605 read 0x72c000 thru 0x735477 (0x9478 bytes) 000305[1] 1788528927.607028 mapwrite 0x475000 thru 0x4760fe (0x10ff bytes) 000306[1] 1788528927.679920 write 0x23b000 thru 0x24338d (0x838e bytes) 000307[1] 1788528927.701151 mapread 0x8f4000 thru 0x8fc5cd (0x85ce bytes) 000308[0] 1788528927.766743 trunc from 0x9f941a to 0x1e1377 (0x8180a2 bytes) 000311[1] 1788528927.919737 write 0x00c000 thru 0x00da88 (0x1a89 bytes) 000312[0] 1788528927.961899 mapwrite 0x09c000 thru 0x0a2081 (0x6082 bytes) 000313[0] 1788528928.070740 punch from 0x094192 to 0x0a2736 (0xe5a5 bytes) 000316[1] 1788528928.093092 trunc from 0x1e1378 to 0x929965 (0x7485ee bytes) 000319[0] 1788528928.160809 trunc from 0x929966 to 0x2c8063 (0x661902 bytes) 000320[1] 1788528928.203187 mapread 0x253000 thru 0x2566ef (0x36f0 bytes) 000322[1] 1788528928.249023 mapread 0x0d6000 thru 0x0d6f09 (0xf0a bytes) 000324[1] 1788528928.258171 write 0x392000 thru 0x39bb01 (0x9b02 bytes) 000325[0] 1788528928.276031 write 0x2d4000 thru 0x2d6e1a (0x2e1b bytes) 000326[1] 1788528928.310245 punch from 0x1b264c to 0x1b7b7c (0x5531 bytes) 000327[0] 1788528928.333254 mapwrite 0x93b000 thru 0x94adad (0xfdae bytes) 000328[1] 1788528928.455224 mapread 0x2d6000 thru 0x2e56a6 (0xf6a7 bytes) 000329[1] 1788528928.519745 mapread 0x2e1000 thru 0x2e489e (0x389f bytes) 000330[0] 1788528928.520348 write 0x8a8000 thru 0x8aabc9 (0x2bca bytes) 000331[0] 1788528928.565179 mapread 0x6c4000 thru 0x6d29be (0xe9bf bytes) 000333[1] 1788528928.592070 read 0x6a7000 thru 0x6b1cc8 (0xacc9 bytes) 000334[0] 1788528928.622602 mapwrite 0x5a7000 thru 0x5b3ca2 (0xcca3 bytes) 000335[1] 1788528928.727771 write 0x5a2000 thru 0x5a5425 (0x3426 bytes) 000337[0] 1788528928.777419 mapwrite 0x07a000 thru 0x07e3e6 (0x43e7 bytes) 000338[0] 1788528928.898342 read 0x36d000 thru 0x37c01c (0xf01d bytes) 000342[1] 1788528928.948367 write 0x593000 thru 0x5970d4 (0x40d5 bytes) 000344[0] 1788528929.022723 read 0x226000 thru 0x22875b (0x275c bytes) 000345[0] 1788528929.057887 punch from 0x2d94b4 to 0x2dc63d (0x318a bytes) 000348[1] 1788528929.104323 trunc from 0x94adae to 0x14a0c1 (0x800cec bytes) 000349[0] 1788528929.151129 trunc from 0x14a0c2 to 0x44091e (0x2f685d bytes) 000351[0] 1788528929.199346 read 0x3a2000 thru 0x3a46dd (0x26de bytes) 000352[1] 1788528929.205881 mapread 0x327000 thru 0x32b07d (0x407e bytes) 000354[1] 1788528929.244781 trunc from 0x44091f to 0x5e6b0a (0x1a61ec bytes) 000355[1] 1788528929.279224 read 0x10d000 thru 0x10dd7e (0xd7f bytes) 000356[0] 1788528929.304200 mapwrite 0x148000 thru 0x14f021 (0x7022 bytes) 000357[1] 1788528929.425966 trunc from 0x5e6b0b to 0x602aaa (0x1bfa0 bytes) 000358[0] 1788528929.469093 mapwrite 0x241000 thru 0x245006 (0x4007 bytes) 000359[1] 1788528929.559695 trunc from 0x602aab to 0x8554e6 (0x252a3c bytes) 000361[1] 1788528929.677241 mapwrite 0x0fa000 thru 0x0fc36a (0x236b bytes) 000362[0] 1788528929.901002 mapread 0x70b000 thru 0x71692c (0xb92d bytes) 000363[1] 1788528929.998887 mapwrite 0x558000 thru 0x562ffa (0xaffb bytes) 000364[0] 1788528930.185746 read 0x145000 thru 0x151346 (0xc347 bytes) 000370[1] 1788528930.228599 mapwrite 0x498000 thru 0x4a41ed (0xc1ee bytes) 000372[0] 1788528930.359485 write 0x0f0000 thru 0x0feb82 (0xeb83 bytes) 000373[1] 1788528930.388031 mapwrite 0x11f000 thru 0x11fb5a (0xb5b bytes) 000374[0] 1788528930.614804 trunc from 0x8554e7 to 0x6d976c (0x17bd7a bytes) 000377[0] 1788528930.686112 write 0x5dc000 thru 0x5e2636 (0x6637 bytes) 000378[0] 1788528930.692105 mapread 0x32f000 thru 0x33a2de (0xb2df bytes) 000379[1] 1788528930.723812 mapread 0x47d000 thru 0x480a44 (0x3a45 bytes) 000382[0] 1788528930.840774 punch from 0x317917 to 0x31f73c (0x7e26 bytes) 000384[1] 1788528930.876714 trunc from 0x6d976d to 0x864837 (0x18b0cb bytes) 000385[0] 1788528930.919803 punch from 0x849fb7 to 0x84cc12 (0x2c5c bytes) 000387[0] 1788528930.955878 trunc from 0x864838 to 0x8d6f9c (0x72765 bytes) 000388[1] 1788528930.992236 write 0x0e7000 thru 0x0f3350 (0xc351 bytes) 000389[1] 1788528931.011114 mapread 0x26c000 thru 0x27be1f (0xfe20 bytes) 000390[1] 1788528931.058972 mapwrite 0x494000 thru 0x499886 (0x5887 bytes) 000391[1] 1788528931.122933 mapwrite 0x823000 thru 0x825a92 (0x2a93 bytes) 000395[1] 1788528931.242562 write 0x3ab000 thru 0x3b6467 (0xb468 bytes) 000396[0] 1788528931.281703 trunc from 0x8d6f9d to 0x829dae (0xad1ee bytes) 000397[0] 1788528931.366430 trunc from 0x829daf to 0x42bd8c (0x3fe022 bytes) 000398[0] 1788528931.408032 read 0x278000 thru 0x280c3f (0x8c40 bytes) 000399[1] 1788528931.493204 write 0x07f000 thru 0x083f3d (0x4f3e bytes) 000402[0] 1788528931.595457 punch from 0x3f20b9 to 0x401c4b (0xfb93 bytes) 000403[1] 1788528931.633655 read 0x189000 thru 0x18d789 (0x478a bytes) 000407[0] 1788528931.713615 write 0x31f000 thru 0x32b534 (0xc535 bytes) 000408[1] 1788528931.746153 punch from 0x3e26e2 to 0x3e389f (0x11be bytes) 000409[1] 1788528931.805196 read 0x088000 thru 0x094068 (0xc069 bytes) 000411[0] 1788528931.820899 write 0x781000 thru 0x784355 (0x3356 bytes) 000413[0] 1788528931.843040 mapwrite 0x431000 thru 0x43918b (0x818c bytes) 000414[1] 1788528931.978676 write 0x1f4000 thru 0x1f6697 (0x2698 bytes) 000415[0] 1788528932.066767 mapwrite 0x128000 thru 0x12ede8 (0x6de9 bytes) 000416[0] 1788528932.205705 punch from 0x759a5c to 0x75cff6 (0x359b bytes) 000418[0] 1788528932.213289 write 0x4da000 thru 0x4dd7ef (0x37f0 bytes) 000419[0] 1788528932.241969 read 0x138000 thru 0x13a7ea (0x27eb bytes) 000421[1] 1788528932.272981 write 0x9bc000 thru 0x9cb30e (0xf30f bytes) 000422[0] 1788528932.307949 mapread 0x1f3000 thru 0x1fca0e (0x9a0f bytes) 000423[1] 1788528932.419408 trunc from 0x9cb30f to 0x3ce794 (0x5fcb7a bytes) 000425[1] 1788528932.559529 punch from 0x2bda5a to 0x2c0c09 (0x31b0 bytes) 000426[1] 1788528932.577861 write 0x225000 thru 0x225ee1 (0xee2 bytes) 000427[1] 1788528932.593228 read 0x2d7000 thru 0x2d9f2b (0x2f2c bytes) 000428[0] 1788528932.733858 mapread 0x2af000 thru 0x2af8f2 (0x8f3 bytes) 000430[1] 1788528932.835825 punch from 0x2a5838 to 0x2a71d0 (0x1999 bytes) 000431[0] 1788528932.863962 read 0x278000 thru 0x279759 (0x175a bytes) 000432[0] 1788528932.995208 mapread 0x23b000 thru 0x24781c (0xc81d bytes) 000433[0] 1788528933.005356 punch from 0x14ceef to 0x14f007 (0x2119 bytes) 000435[0] 1788528933.049928 trunc from 0x3ce795 to 0x1fdb96 (0x1d0bfe bytes) 000438[1] 1788528933.094771 punch from 0x0ea790 to 0x0efb80 (0x53f1 bytes) 000439[1] 1788528933.128224 trunc from 0x1fdb97 to 0x6e19e1 (0x4e3e4b bytes) 000440[0] 1788528933.172377 read 0x2c2000 thru 0x2c6673 (0x4674 bytes) 000441[1] 1788528933.198325 trunc from 0x6e19e2 to 0x1559a1 (0x58c040 bytes) 000443[1] 1788528933.254028 mapread 0x033000 thru 0x0337b8 (0x7b9 bytes) 000446[1] 1788528933.273039 mapwrite 0x193000 thru 0x19d265 (0xa266 bytes) 000448[0] 1788528933.501332 punch from 0x0b91e8 to 0x0bdd78 (0x4b91 bytes) 000449[0] 1788528933.544225 mapread 0x0a6000 thru 0x0b28bc (0xc8bd bytes) 000450[1] 1788528933.645680 read 0x123000 thru 0x12d515 (0xa516 bytes) 000451[1] 1788528933.718507 mapread 0x0a6000 thru 0x0afe5b (0x9e5c bytes) 000452[1] 1788528933.719386 punch from 0x010d5b to 0x01966b (0x8911 bytes) 000453[1] 1788528933.755792 write 0x4ec000 thru 0x4fb070 (0xf071 bytes) 000455[0] 1788528933.760618 read 0x08b000 thru 0x09510a (0xa10b bytes) 000456[0] 1788528933.894934 read 0x2a4000 thru 0x2a608b (0x208c bytes) 000457[1] 1788528933.910948 mapwrite 0x433000 thru 0x43788e (0x488f bytes) 000458[1] 1788528934.122979 write 0x966000 thru 0x96c067 (0x6068 bytes) 000459[1] 1788528934.125332 punch from 0x503d06 to 0x50d0fd (0x93f8 bytes) 000462[0] 1788528934.147714 mapwrite 0x3aa000 thru 0x3b57d1 (0xb7d2 bytes) 000464[0] 1788528934.322422 mapread 0x747000 thru 0x74c4ea (0x54eb bytes) 000467[1] 1788528934.363423 write 0x93e000 thru 0x9493b6 (0xb3b7 bytes) 000470[1] 1788528934.482845 mapwrite 0x303000 thru 0x303a82 (0xa83 bytes) 000472[0] 1788528934.544030 mapread 0x18f000 thru 0x19e779 (0xf77a bytes) 000473[0] 1788528934.639837 write 0x6a5000 thru 0x6a8929 (0x392a bytes) 000474[1] 1788528934.659759 punch from 0x9062c9 to 0x908a06 (0x273e bytes) 000476[1] 1788528934.719646 mapwrite 0x5cd000 thru 0x5d49dd (0x79de bytes) 000477[0] 1788528934.874510 read 0x5d6000 thru 0x5e2903 (0xc904 bytes) 000478[0] 1788528934.916230 punch from 0x85a53c to 0x85a85d (0x322 bytes) 000480[0] 1788528934.937433 trunc from 0x96c068 to 0x132374 (0x839cf3 bytes) 000484[0] 1788528935.173781 trunc from 0x132375 to 0x6f975c (0x5c73e8 bytes) 000485[0] 1788528935.214894 mapread 0x127000 thru 0x134408 (0xd409 bytes) 000486[0] 1788528935.252840 mapread 0x281000 thru 0x28ee30 (0xde31 bytes) 000487[1] 1788528935.294802 mapread 0x6d6000 thru 0x6e4ec1 (0xeec2 bytes) 000488[1] 1788528935.507685 read 0x1f5000 thru 0x1f6ae1 (0x1ae2 bytes) 000490[1] 1788528935.534958 write 0x994000 thru 0x9952a6 (0x12a7 bytes) 000491[0] 1788528935.548579 trunc from 0x9952a7 to 0x743cb2 (0x2515f4 bytes) 000492[0] 1788528935.594937 write 0x6a6000 thru 0x6aae07 (0x4e08 bytes) 000493[0] 1788528935.621596 mapread 0x078000 thru 0x080568 (0x8569 bytes) 000494[0] 1788528935.655395 read 0x703000 thru 0x70a07b (0x707c bytes) 000497[1] 1788528935.745062 punch from 0x4d211a to 0x4d550f (0x33f6 bytes) 000498[1] 1788528935.779319 write 0x3db000 thru 0x3e5c30 (0xac31 bytes) 000499[0] 1788528935.792108 punch from 0x470418 to 0x4793d3 (0x8fbc bytes) 000500[1] 1788528935.827606 write 0x25f000 thru 0x26e1b3 (0xf1b4 bytes) All operations completed A-OK! PASS 16a (59s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 09:35:41 (1788528941) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.517153 s, 16.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.402096 s, 20.9 MB/s Chance of close/open is 1 in 50 Seed set to 8945 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[1] 1788528946.665758 mapwrite 0x4b0e04 thru 0x4b2c63 (0x1e60 bytes) truncating to largest ever: 0x12f4ab 000004[1] 1788528946.776111 trunc from 0x4b2c64 to 0x12f4aa (0x3837b9 bytes) 000006[1] 1788528946.824815 mapread 0x0ade87 thru 0x0ae14a (0x2c4 bytes) 000007[0] 1788528946.869875 mapread 0x0d3fcb thru 0x0d5342 (0x1378 bytes) 000008[1] 1788528946.930250 write 0x29cc34 thru 0x29cef4 (0x2c1 bytes) 000009[1] 1788528946.945241 write 0x1e87d9 thru 0x1e8c73 (0x49b bytes) 000011[0] 1788528947.002983 write 0x4cc891 thru 0x4cdf4e (0x16be bytes) 000014[0] 1788528947.097088 read 0x21dc12 thru 0x21e4f1 (0x8e0 bytes) 000017[1] 1788528947.129375 punch from 0x3a2541 to 0x3a333a (0xdfa bytes) 000018[1] 1788528947.224751 read 0x141feb thru 0x143bbf (0x1bd5 bytes) 000019[0] 1788528947.273920 write 0x067caa thru 0x068cad (0x1004 bytes) 000020[0] 1788528947.303075 mapread 0x318cd9 thru 0x3199bc (0xce4 bytes) 000022[0] 1788528947.405293 write 0x4c4511 thru 0x4c5176 (0xc66 bytes) 000023[0] 1788528947.467987 punch from 0x153ac1 to 0x15467b (0xbbb bytes) 000024[0] 1788528947.488887 trunc from 0x4cdf4f to 0x049600 (0x48494e bytes) 000026[0] 1788528947.568076 mapwrite 0x73ac44 thru 0x73b171 (0x52e bytes) truncating to largest ever: 0x9bfbda 000027[1] 1788528947.809472 trunc from 0x73b172 to 0x9bfbd9 (0x284a68 bytes) 000028[0] 1788528947.863882 write 0x906590 thru 0x908247 (0x1cb8 bytes) 000030[1] 1788528947.933235 write 0x449715 thru 0x44a5e0 (0xecc bytes) 000031[0] 1788528947.964245 mapwrite 0x09d828 thru 0x09e0ee (0x8c7 bytes) 000032[0] 1788528948.088716 punch from 0x0dee7a to 0x0dfcd2 (0xe59 bytes) 000033[1] 1788528948.112163 mapread 0x5624ec thru 0x563240 (0xd55 bytes) 000034[0] 1788528948.148483 punch from 0x0e9169 to 0x0e9b74 (0xa0c bytes) 000036[0] 1788528948.197896 punch from 0x7061e6 to 0x706779 (0x594 bytes) 000037[0] 1788528948.247661 trunc from 0x9bfbda to 0x3996fa (0x6264df bytes) 000038[0] 1788528948.276545 mapread 0x255793 thru 0x2569ca (0x1238 bytes) 000042[1] 1788528948.372191 write 0x03b102 thru 0x03c374 (0x1273 bytes) 000046[0] 1788528948.455715 trunc from 0x3996fb to 0x1c6bff (0x1d2afb bytes) 000047[0] 1788528948.495709 trunc from 0x1c6c00 to 0x687a48 (0x4c0e49 bytes) 000051[0] 1788528948.661084 mapwrite 0x2d59f7 thru 0x2d79ce (0x1fd8 bytes) 000055[0] 1788528948.990435 trunc from 0x687a49 to 0x4ce25b (0x1b97ed bytes) 000056[0] 1788528949.100077 mapread 0x053654 thru 0x054661 (0x100e bytes) 000057[1] 1788528949.139489 mapread 0x468255 thru 0x469c7f (0x1a2b bytes) 000058[0] 1788528949.207179 write 0x9928f1 thru 0x992c63 (0x373 bytes) 000061[1] 1788528949.244813 punch from 0x8a1ae7 to 0x8a3760 (0x1c7a bytes) 000062[0] 1788528949.343651 mapwrite 0x797401 thru 0x7979a6 (0x5a6 bytes) 000064[0] 1788528949.588315 mapwrite 0x34e362 thru 0x35003b (0x1cda bytes) 000065[0] 1788528949.783919 punch from 0x1b5838 to 0x1b632e (0xaf7 bytes) 000066[0] 1788528949.792533 trunc from 0x992c64 to 0x31c715 (0x67654e bytes) 000067[0] 1788528949.820826 trunc from 0x31c716 to 0x2281a2 (0xf4573 bytes) 000068[1] 1788528949.859787 write 0x3e8c0e thru 0x3ea78d (0x1b80 bytes) 000069[1] 1788528949.900803 mapwrite 0x758c5d thru 0x7591ba (0x55e bytes) 000070[1] 1788528949.998733 read 0x74888b thru 0x749a24 (0x119a bytes) 000071[1] 1788528950.012948 punch from 0x314ab7 to 0x316778 (0x1cc2 bytes) 000072[1] 1788528950.100096 trunc from 0x7591bb to 0x930783 (0x1d75c9 bytes) 000073[1] 1788528950.132019 mapread 0x73573a thru 0x73661e (0xee5 bytes) 000074[1] 1788528950.206162 write 0x613f4a thru 0x6142cb (0x382 bytes) 000078[1] 1788528950.277562 read 0x7f32d9 thru 0x7f4380 (0x10a8 bytes) 000079[1] 1788528950.285839 trunc from 0x930784 to 0x43aac7 (0x4f5cbc bytes) 000080[0] 1788528950.317193 write 0x062a73 thru 0x0631ea (0x778 bytes) 000082[0] 1788528950.401527 trunc from 0x43aac8 to 0x7e6b2e (0x3ac067 bytes) 000083[0] 1788528950.462806 mapwrite 0x6f0498 thru 0x6f0d4d (0x8b6 bytes) 000084[0] 1788528950.536518 mapwrite 0x7cee58 thru 0x7d0533 (0x16dc bytes) 000086[0] 1788528950.646431 read 0x4d100e thru 0x4d11b4 (0x1a7 bytes) 000087[0] 1788528950.671119 trunc from 0x7e6b2f to 0x99bf42 (0x1b5414 bytes) 000090[0] 1788528950.768970 read 0x47641b thru 0x477951 (0x1537 bytes) 000091[1] 1788528950.784824 write 0x053fee thru 0x0540fc (0x10f bytes) 000092[1] 1788528950.854750 write 0x80e207 thru 0x80fbcd (0x19c7 bytes) 000093[0] 1788528950.882306 punch from 0x808d64 to 0x808def (0x8c bytes) 000094[0] 1788528950.933547 mapwrite 0x47547a thru 0x475551 (0xd8 bytes) 000095[1] 1788528951.036382 mapwrite 0x42020f thru 0x422165 (0x1f57 bytes) 000096[1] 1788528951.229730 write 0x60ad79 thru 0x60bda4 (0x102c bytes) 000098[0] 1788528951.294248 mapread 0x068189 thru 0x069241 (0x10b9 bytes) 000100[0] 1788528951.354770 write 0x994541 thru 0x995092 (0xb52 bytes) 000102[1] 1788528951.402382 write 0x585bf9 thru 0x586567 (0x96f bytes) 000106[1] 1788528951.486357 write 0x005589 thru 0x00596d (0x3e5 bytes) 000107[0] 1788528951.515494 trunc from 0x99bf43 to 0x6f457c (0x2a79c6 bytes) 000108[1] 1788528951.612663 mapread 0x6db81b thru 0x6dd6e7 (0x1ecd bytes) 000109[1] 1788528951.675599 trunc from 0x6f457d to 0x1d0b32 (0x523a4a bytes) 000111[1] 1788528951.745977 punch from 0x0d9447 to 0x0d9c36 (0x7f0 bytes) 000113[1] 1788528951.775869 mapread 0x170890 thru 0x171ece (0x163f bytes) 000115[0] 1788528951.810459 mapread 0x085dfd thru 0x08707e (0x1282 bytes) 000116[0] 1788528951.866947 read 0x0c59ec thru 0x0c5ebe (0x4d3 bytes) 000118[1] 1788528952.026158 write 0x08a8e8 thru 0x08b911 (0x102a bytes) 000119[0] 1788528952.077549 write 0x9510f0 thru 0x951521 (0x432 bytes) 000121[1] 1788528952.174416 write 0x4931cc thru 0x4933d9 (0x20e bytes) 000124[1] 1788528952.254787 write 0x899f75 thru 0x89a04e (0xda bytes) 000126[1] 1788528952.289928 punch from 0x8826cd to 0x8845db (0x1f0f bytes) 000128[1] 1788528952.300000 mapread 0x2072a3 thru 0x2080fd (0xe5b bytes) 000129[0] 1788528952.318882 mapwrite 0x742396 thru 0x74274c (0x3b7 bytes) 000132[1] 1788528952.475743 mapread 0x008251 thru 0x0087e7 (0x597 bytes) 000133[1] 1788528952.512381 write 0x4e1efe thru 0x4e2bbb (0xcbe bytes) 000135[1] 1788528952.567883 trunc from 0x951522 to 0x1102f1 (0x841230 bytes) 000136[0] 1788528952.623900 trunc from 0x1102f2 to 0x0eba96 (0x2485b bytes) 000137[1] 1788528952.721763 trunc from 0x0eba97 to 0x6f1123 (0x60568d bytes) 000138[0] 1788528952.855466 trunc from 0x6f1124 to 0x8671c8 (0x1760a5 bytes) 000139[0] 1788528952.937340 mapwrite 0x965f8e thru 0x966f91 (0x1004 bytes) 000141[1] 1788528953.317231 trunc from 0x966f92 to 0x6b5ca8 (0x2b12e9 bytes) 000142[0] 1788528953.457389 mapread 0x3987a5 thru 0x39a412 (0x1c6e bytes) 000143[1] 1788528953.506672 read 0x4e1a77 thru 0x4e2e0d (0x1397 bytes) 000144[0] 1788528953.528063 mapread 0x6368cc thru 0x637236 (0x96b bytes) 000146[0] 1788528953.566376 punch from 0x62a868 to 0x62c594 (0x1d2d bytes) 000147[1] 1788528953.637143 read 0x6679b7 thru 0x66950b (0x1b55 bytes) 000149[1] 1788528953.677634 read 0x0df239 thru 0x0dfa26 (0x7ee bytes) 000152[0] 1788528953.691665 mapwrite 0x23b65e thru 0x23c1ee (0xb91 bytes) 000153[1] 1788528953.910870 punch from 0x077a2a to 0x0791aa (0x1781 bytes) 000156[1] 1788528953.985110 trunc from 0x6b5ca9 to 0x884bb3 (0x1cef0b bytes) 000157[1] 1788528954.037688 mapwrite 0x8cdae0 thru 0x8cf2d3 (0x17f4 bytes) 000159[1] 1788528954.298829 punch from 0x550eb5 to 0x55259d (0x16e9 bytes) 000161[1] 1788528954.369809 mapwrite 0x47be04 thru 0x47c485 (0x682 bytes) 000162[1] 1788528954.501976 read 0x40350b thru 0x4052d2 (0x1dc8 bytes) 000163[0] 1788528954.545426 mapwrite 0x59c7c0 thru 0x59d59d (0xdde bytes) 000164[1] 1788528954.675918 punch from 0x1a2739 to 0x1a2cad (0x575 bytes) 000168[0] 1788528954.821122 trunc from 0x8cf2d4 to 0x47cddc (0x4524f7 bytes) 000169[0] 1788528955.026400 mapread 0x0f4ef4 thru 0x0f61f8 (0x1305 bytes) 000170[1] 1788528955.119252 trunc from 0x47cddd to 0x8bcf26 (0x44014a bytes) 000171[0] 1788528955.286877 write 0x5fde30 thru 0x5feab2 (0xc83 bytes) 000172[1] 1788528955.354178 read 0x08392d thru 0x083c8f (0x363 bytes) 000173[0] 1788528955.419461 mapread 0x2c5eed thru 0x2c7b48 (0x1c5c bytes) 000179[0] 1788528955.547468 read 0x29b9ab thru 0x29bcb9 (0x30f bytes) 000180[1] 1788528955.570087 mapwrite 0x4dbdbb thru 0x4dca7a (0xcc0 bytes) 000181[1] 1788528955.721336 write 0x4deeda thru 0x4df383 (0x4aa bytes) 000183[0] 1788528955.755827 punch from 0x5738f5 to 0x57462d (0xd39 bytes) 000184[0] 1788528955.796944 mapwrite 0x7e9730 thru 0x7ea237 (0xb08 bytes) 000185[0] 1788528955.870231 mapread 0x1dc2c4 thru 0x1de081 (0x1dbe bytes) 000186[1] 1788528955.892846 mapwrite 0x70fe75 thru 0x710582 (0x70e bytes) 000187[0] 1788528956.031786 trunc from 0x8bcf27 to 0x6c6e85 (0x1f60a1 bytes) 000188[1] 1788528956.157771 mapread 0x446ab7 thru 0x448273 (0x17bd bytes) 000189[1] 1788528956.242769 punch from 0x334852 to 0x3356d4 (0xe83 bytes) 000190[0] 1788528956.287765 punch from 0x2a409b to 0x2a5588 (0x14ee bytes) 000191[1] 1788528956.319787 mapwrite 0x135fc2 thru 0x1371d6 (0x1215 bytes) 000193[1] 1788528956.462066 write 0x13b9b5 thru 0x13bfde (0x62a bytes) 000196[1] 1788528956.545723 punch from 0x322b3d to 0x324582 (0x1a46 bytes) 000197[0] 1788528956.592397 write 0x5e17ca thru 0x5e2895 (0x10cc bytes) 000199[0] 1788528956.758237 mapread 0x37df17 thru 0x37e56b (0x655 bytes) 000200[1] 1788528956.867643 trunc from 0x6c6e86 to 0x4080d2 (0x2bedb3 bytes) 000201[0] 1788528957.002921 mapwrite 0x5def7d thru 0x5df698 (0x71c bytes) 000202[1] 1788528957.197752 punch from 0x58252e to 0x583dab (0x187e bytes) 000206[0] 1788528957.342970 read 0x483116 thru 0x4845ec (0x14d7 bytes) 000207[1] 1788528957.400634 trunc from 0x5df699 to 0x64b927 (0x6c28f bytes) 000208[0] 1788528957.456673 punch from 0x1ab53d to 0x1acd89 (0x184d bytes) 000209[0] 1788528957.502185 mapwrite 0x62f9f4 thru 0x631200 (0x180d bytes) 000210[0] 1788528957.630678 mapread 0x2f4d98 thru 0x2f670b (0x1974 bytes) 000211[0] 1788528957.656695 trunc from 0x64b928 to 0x8025bd (0x1b6c96 bytes) 000213[0] 1788528957.730294 read 0x656756 thru 0x6585d4 (0x1e7f bytes) 000214[0] 1788528957.738555 write 0x50076c thru 0x501677 (0xf0c bytes) 000215[0] 1788528957.751203 read 0x2246a3 thru 0x225266 (0xbc4 bytes) 000216[1] 1788528957.780501 punch from 0x207f10 to 0x209143 (0x1234 bytes) 000218[0] 1788528957.834919 mapwrite 0x8f2c82 thru 0x8f339c (0x71b bytes) 000219[0] 1788528957.960509 mapread 0x50d28b thru 0x50ee4b (0x1bc1 bytes) 000226[0] 1788528958.217757 punch from 0x72c11d to 0x72c5ee (0x4d2 bytes) 000228[1] 1788528958.229779 punch from 0x733410 to 0x734a60 (0x1651 bytes) 000229[0] 1788528958.286337 read 0x856633 thru 0x857a90 (0x145e bytes) 000234[0] 1788528958.414775 mapwrite 0x86de44 thru 0x86f361 (0x151e bytes) 000235[1] 1788528958.561115 write 0x0c83e1 thru 0x0c97d7 (0x13f7 bytes) 000236[1] 1788528958.589316 mapread 0x324ff8 thru 0x32542e (0x437 bytes) 000238[1] 1788528958.630083 trunc from 0x8f339d to 0x371fd3 (0x5813c9 bytes) 000239[1] 1788528958.659406 mapwrite 0x1ac991 thru 0x1ad453 (0xac3 bytes) 000240[0] 1788528958.731009 punch from 0x041178 to 0x042828 (0x16b1 bytes) 000243[0] 1788528958.870223 write 0x7d3b1d thru 0x7d4865 (0xd49 bytes) 000245[1] 1788528959.004775 trunc from 0x7d4866 to 0x37c5ab (0x4582ba bytes) 000248[0] 1788528959.140148 read 0x0c6355 thru 0x0c73a1 (0x104d bytes) 000251[1] 1788528959.327794 punch from 0x0cc80a to 0x0ccff7 (0x7ee bytes) 000253[0] 1788528959.413546 write 0x342acf thru 0x342cc0 (0x1f2 bytes) 000254[1] 1788528959.477149 punch from 0x0a865e to 0x0a8aa2 (0x445 bytes) 000257[0] 1788528959.605667 read 0x05b413 thru 0x05ca1f (0x160d bytes) 000258[0] 1788528959.845334 write 0x7d54d1 thru 0x7d5903 (0x433 bytes) 000261[1] 1788528959.917514 trunc from 0x7d5904 to 0x998e3c (0x1c3539 bytes) 000262[1] 1788528959.977437 read 0x64b5b4 thru 0x64bb33 (0x580 bytes) 000263[1] 1788528960.017107 punch from 0x42ac3b to 0x42c084 (0x144a bytes) 000264[0] 1788528960.058208 mapwrite 0x7ccfd8 thru 0x7cef16 (0x1f3f bytes) 000268[1] 1788528960.257477 trunc from 0x998e3d to 0x78e3e7 (0x20aa55 bytes) 000269[1] 1788528960.340943 mapwrite 0x80e5bc thru 0x80fedb (0x1920 bytes) 000270[0] 1788528960.566145 write 0x3ea38d thru 0x3eab02 (0x776 bytes) 000271[0] 1788528960.630443 write 0x879030 thru 0x87a817 (0x17e8 bytes) 000272[0] 1788528960.659836 read 0x471a62 thru 0x47262c (0xbcb bytes) 000273[0] 1788528960.679067 punch from 0x311ef3 to 0x313476 (0x1584 bytes) 000274[1] 1788528960.718651 mapwrite 0x379f87 thru 0x37a5c6 (0x640 bytes) 000275[1] 1788528960.857799 trunc from 0x87a818 to 0x8770b4 (0x3763 bytes) 000279[0] 1788528961.033310 mapwrite 0x1fef0c thru 0x1ff879 (0x96e bytes) 000280[1] 1788528961.178515 mapread 0x1aa98b thru 0x1ac6d0 (0x1d46 bytes) 000281[0] 1788528961.232825 write 0x30e527 thru 0x30e9fe (0x4d8 bytes) 000283[1] 1788528961.332804 mapwrite 0x3f684a thru 0x3f784d (0x1004 bytes) 000284[0] 1788528961.504442 write 0x048070 thru 0x0489b8 (0x949 bytes) 000285[1] 1788528961.543366 write 0x3834ff thru 0x38408a (0xb8c bytes) 000286[1] 1788528961.599516 trunc from 0x8770b5 to 0x03aeea (0x83c1ca bytes) 000287[0] 1788528961.669383 read 0x0109b1 thru 0x0122c8 (0x1918 bytes) 000288[1] 1788528961.730913 read 0x009589 thru 0x00a1a9 (0xc21 bytes) 000289[1] 1788528961.783050 trunc from 0x03aeeb to 0x0ac568 (0x7167e bytes) 000291[1] 1788528961.883253 mapread 0x095b3e thru 0x096a22 (0xee5 bytes) 000292[1] 1788528961.959409 read 0x0a355c thru 0x0a3cb2 (0x757 bytes) 000294[1] 1788528962.007814 trunc from 0x0ac569 to 0x645604 (0x59909c bytes) 000295[1] 1788528962.087792 trunc from 0x645605 to 0x21aa58 (0x42abac bytes) 000296[0] 1788528962.200617 mapread 0x10f12f thru 0x10fa28 (0x8fa bytes) 000298[0] 1788528962.420144 write 0x4b51e9 thru 0x4b7033 (0x1e4b bytes) 000299[1] 1788528962.455071 mapwrite 0x937aa4 thru 0x93812b (0x688 bytes) 000301[1] 1788528962.905335 read 0x0717d6 thru 0x073653 (0x1e7e bytes) 000302[0] 1788528962.983705 mapwrite 0x44e0a1 thru 0x44f83d (0x179d bytes) 000304[0] 1788528963.163403 punch from 0x1f7958 to 0x1f7d7f (0x428 bytes) truncating to largest ever: 0x9fe0ba 000306[1] 1788528963.219389 trunc from 0x93812c to 0x9fe0b9 (0xc5f8e bytes) 000307[0] 1788528963.316844 mapwrite 0x210805 thru 0x2124f0 (0x1cec bytes) 000312[0] 1788528963.697333 mapread 0x7dcb1a thru 0x7dd71e (0xc05 bytes) 000313[1] 1788528963.879152 mapwrite 0x85af00 thru 0x85b449 (0x54a bytes) 000315[0] 1788528964.079231 punch from 0x4b6f09 to 0x4b73e6 (0x4de bytes) 000319[1] 1788528964.225168 trunc from 0x9fe0ba to 0x6a28c0 (0x35b7f9 bytes) 000320[0] 1788528964.287462 read 0x5b9e72 thru 0x5ba675 (0x804 bytes) 000325[1] 1788528964.449695 read 0x263839 thru 0x2657b3 (0x1f7b bytes) 000327[1] 1788528964.484019 mapread 0x062cd0 thru 0x062f1e (0x24f bytes) 000328[1] 1788528964.492404 write 0x254ad7 thru 0x254fff (0x529 bytes) 000329[1] 1788528964.499745 write 0x918c3a thru 0x9197c1 (0xb88 bytes) 000330[0] 1788528964.502741 mapwrite 0x054e11 thru 0x056a6e (0x1c5e bytes) 000331[0] 1788528964.648564 write 0x371b4c thru 0x373101 (0x15b6 bytes) 000332[0] 1788528964.661687 read 0x84b9ea thru 0x84c4c3 (0xada bytes) 000334[1] 1788528964.670521 read 0x3e071d thru 0x3e1a5d (0x1341 bytes) 000335[0] 1788528964.703306 mapwrite 0x437f4d thru 0x439d39 (0x1ded bytes) 000336[0] 1788528964.822963 punch from 0x0ffa36 to 0x101643 (0x1c0e bytes) 000338[0] 1788528964.874639 read 0x26a4dc thru 0x26b7f7 (0x131c bytes) 000339[0] 1788528964.960238 mapwrite 0x957ce0 thru 0x959492 (0x17b3 bytes) 000340[0] 1788528965.318983 write 0x6acd8f thru 0x6ad813 (0xa85 bytes) 000341[1] 1788528965.406132 mapwrite 0x74f148 thru 0x75040a (0x12c3 bytes) 000342[1] 1788528965.571120 mapread 0x492f31 thru 0x4946de (0x17ae bytes) 000344[1] 1788528965.667243 mapwrite 0x191918 thru 0x1929ff (0x10e8 bytes) 000345[0] 1788528965.794954 mapread 0x8cec20 thru 0x8cf735 (0xb16 bytes) 000347[0] 1788528965.878139 punch from 0x70fd13 to 0x7114df (0x17cd bytes) 000348[1] 1788528965.924162 trunc from 0x959493 to 0x7e0548 (0x178f4a bytes) 000350[0] 1788528966.060709 mapread 0x2804c8 thru 0x2811a2 (0xcdb bytes) 000352[1] 1788528966.124479 mapread 0x7e033a thru 0x7e0548 (0x20f bytes) 000353[1] 1788528966.153790 mapwrite 0x048f37 thru 0x04938c (0x456 bytes) 000355[1] 1788528966.298001 mapread 0x47ed6e thru 0x47feca (0x115d bytes) 000356[1] 1788528966.311468 mapread 0x033ba0 thru 0x033ed5 (0x336 bytes) 000357[0] 1788528966.347413 mapwrite 0x3c0bd3 thru 0x3c221c (0x164a bytes) 000358[1] 1788528966.504497 mapwrite 0x8e33c5 thru 0x8e4986 (0x15c2 bytes) 000361[0] 1788528966.617883 mapwrite 0x5618d9 thru 0x56364a (0x1d72 bytes) 000362[1] 1788528966.751418 trunc from 0x8e4987 to 0x8e1529 (0x345d bytes) 000363[1] 1788528966.837155 trunc from 0x8e152a to 0x2f5b0a (0x5eba1f bytes) 000364[0] 1788528966.866945 trunc from 0x2f5b0b to 0x2efb7d (0x5f8d bytes) 000366[0] 1788528967.021596 mapwrite 0x04bbcf thru 0x04c755 (0xb87 bytes) 000368[1] 1788528967.145912 trunc from 0x2efb7e to 0x3ddf34 (0xee3b7 bytes) 000369[0] 1788528967.226611 read 0x1fdf79 thru 0x1fefa9 (0x1031 bytes) 000370[0] 1788528967.363202 mapwrite 0x5c3641 thru 0x5c3b69 (0x529 bytes) 000372[1] 1788528967.627091 punch from 0x4387b9 to 0x439996 (0x11de bytes) 000374[1] 1788528967.692899 read 0x1a02ba thru 0x1a0aa8 (0x7ef bytes) 000375[0] 1788528967.746522 read 0x51e034 thru 0x51ea8f (0xa5c bytes) 000378[1] 1788528967.784538 punch from 0x5b185a to 0x5b21e5 (0x98c bytes) 000380[1] 1788528967.847612 write 0x349495 thru 0x34a621 (0x118d bytes) 000382[0] 1788528967.958472 write 0x09ec29 thru 0x09f754 (0xb2c bytes) 000384[1] 1788528968.050693 write 0x0a41f4 thru 0x0a4fb4 (0xdc1 bytes) 000386[0] 1788528968.144756 trunc from 0x5c3b6a to 0x285265 (0x33e904 bytes) 000388[1] 1788528968.264657 trunc from 0x285266 to 0x314942 (0x8f6dd bytes) 000389[0] 1788528968.364720 read 0x16216f thru 0x16369f (0x1531 bytes) 000392[0] 1788528968.543045 write 0x122ef4 thru 0x124d75 (0x1e82 bytes) 000393[0] 1788528968.612478 trunc from 0x314943 to 0x7f52cd (0x4e098b bytes) 000394[1] 1788528968.670046 read 0x0d74a5 thru 0x0d8a6c (0x15c8 bytes) 000395[0] 1788528968.793600 trunc from 0x7f52ce to 0x874e25 (0x7fb58 bytes) 000396[1] 1788528968.896095 mapread 0x0393bb thru 0x03acf9 (0x193f bytes) 000397[0] 1788528968.962816 mapwrite 0x6add12 thru 0x6af277 (0x1566 bytes) 000399[0] 1788528969.191156 write 0x098506 thru 0x0987d2 (0x2cd bytes) 000401[0] 1788528969.278371 punch from 0x1581a3 to 0x159af8 (0x1956 bytes) 000402[0] 1788528969.301661 punch from 0x8162aa to 0x8168fc (0x653 bytes) 000403[1] 1788528969.363161 write 0x1aabf4 thru 0x1ab8cc (0xcd9 bytes) 000404[1] 1788528969.455725 mapread 0x1ca719 thru 0x1cb59f (0xe87 bytes) 000406[1] 1788528969.528428 read 0x56c9c3 thru 0x56ded1 (0x150f bytes) 000407[0] 1788528969.554061 read 0x46e22b thru 0x46eb35 (0x90b bytes) 000409[0] 1788528969.565937 mapread 0x257044 thru 0x2571b6 (0x173 bytes) 000413[1] 1788528969.615108 mapwrite 0x4fd20b thru 0x4fd937 (0x72d bytes) 000415[0] 1788528969.744370 write 0x9feaa2 thru 0x9ffe46 (0x13a5 bytes) 000416[0] 1788528969.775493 punch from 0x103ce8 to 0x104dd2 (0x10eb bytes) 000417[1] 1788528969.800540 trunc from 0x9ffe47 to 0x7ad647 (0x2527ff bytes) 000418[1] 1788528969.879344 punch from 0x5284e3 to 0x5285d8 (0xf6 bytes) 000420[0] 1788528969.928538 mapwrite 0x7bdc43 thru 0x7be0a1 (0x45f bytes) 000421[1] 1788528970.064559 mapwrite 0x51f11b thru 0x51f2fa (0x1e0 bytes) 000423[0] 1788528970.213053 write 0x2f1299 thru 0x2f2a72 (0x17da bytes) 000424[1] 1788528970.246246 read 0x516eb1 thru 0x517561 (0x6b1 bytes) 000426[1] 1788528970.290480 trunc from 0x7be0a2 to 0x142621 (0x67ba80 bytes) 000428[0] 1788528970.412991 read 0x1374f0 thru 0x138564 (0x1075 bytes) 000429[0] 1788528970.520400 punch from 0x07238a to 0x073747 (0x13be bytes) 000431[1] 1788528970.591161 punch from 0x1110e8 to 0x112ec3 (0x1ddc bytes) 000432[1] 1788528970.611350 read 0x0b064e thru 0x0b204c (0x19ff bytes) 000434[1] 1788528970.757510 trunc from 0x142622 to 0x51aab0 (0x3d848f bytes) 000435[0] 1788528970.797624 read 0x324897 thru 0x32661a (0x1d84 bytes) 000436[0] 1788528970.947868 punch from 0x437d84 to 0x438759 (0x9d6 bytes) 000440[1] 1788528971.051496 trunc from 0x51aab1 to 0x2e07f0 (0x23a2c0 bytes) 000445[1] 1788528971.297052 trunc from 0x2e07f1 to 0x2658b8 (0x7af38 bytes) 000446[0] 1788528971.346973 punch from 0x055609 to 0x056116 (0xb0e bytes) 000447[1] 1788528971.395497 read 0x036886 thru 0x03836a (0x1ae5 bytes) 000448[1] 1788528971.476498 punch from 0x241072 to 0x242d12 (0x1ca1 bytes) 000449[1] 1788528971.539083 trunc from 0x2658b9 to 0x4bcfbb (0x257703 bytes) 000450[1] 1788528971.566440 write 0x157c4e thru 0x1585e0 (0x993 bytes) 000451[1] 1788528971.613237 punch from 0x1c1199 to 0x1c2ef8 (0x1d60 bytes) 000452[1] 1788528971.675202 mapwrite 0x2bd411 thru 0x2be86d (0x145d bytes) 000453[0] 1788528971.735883 write 0x2013b1 thru 0x202ffb (0x1c4b bytes) 000454[1] 1788528971.793221 punch from 0x17286e to 0x17476f (0x1f02 bytes) 000455[1] 1788528971.865352 mapread 0x2399f9 thru 0x23b30c (0x1914 bytes) 000458[0] 1788528971.970889 mapread 0x414f6d thru 0x416562 (0x15f6 bytes) 000459[0] 1788528972.056969 write 0x056f93 thru 0x057c87 (0xcf5 bytes) 000460[1] 1788528972.173535 mapwrite 0x55927e thru 0x5596b2 (0x435 bytes) 000462[0] 1788528972.384000 write 0x1a79d0 thru 0x1a957b (0x1bac bytes) 000463[0] 1788528972.444036 read 0x177425 thru 0x178cdb (0x18b7 bytes) 000464[0] 1788528972.472103 punch from 0x04d083 to 0x04d20e (0x18c bytes) 000465[1] 1788528972.500968 mapread 0x35438e thru 0x356328 (0x1f9b bytes) 000466[0] 1788528972.582952 trunc from 0x5596b3 to 0x11e0ef (0x43b5c3 bytes) 000468[1] 1788528972.763027 mapread 0x1157fc thru 0x116832 (0x1037 bytes) 000470[1] 1788528972.831921 trunc from 0x11e0f0 to 0x796f57 (0x678e68 bytes) 000471[1] 1788528972.888506 mapread 0x2a9340 thru 0x2ab285 (0x1f46 bytes) 000473[0] 1788528972.908254 punch from 0x72f48d to 0x72fef7 (0xa6b bytes) 000477[1] 1788528973.121163 mapread 0x365d53 thru 0x3667f7 (0xaa5 bytes) 000479[0] 1788528973.214222 mapread 0x68d3d0 thru 0x68da0f (0x640 bytes) 000480[0] 1788528973.264749 read 0x6293cb thru 0x62b018 (0x1c4e bytes) 000481[0] 1788528973.295892 write 0x7b1eba thru 0x7b2630 (0x777 bytes) 000482[0] 1788528973.314038 mapread 0x135f34 thru 0x1379d9 (0x1aa6 bytes) 000483[0] 1788528973.371870 punch from 0x67259c to 0x672ba7 (0x60c bytes) 000484[1] 1788528973.398731 punch from 0x3f1b0a to 0x3f3615 (0x1b0c bytes) 000485[1] 1788528973.450572 punch from 0x38c125 to 0x38d0c2 (0xf9e bytes) 000487[0] 1788528973.484940 punch from 0x60bded to 0x60ce70 (0x1084 bytes) 000489[0] 1788528973.512495 punch from 0x2dd4ea to 0x2dd8d2 (0x3e9 bytes) 000490[0] 1788528973.532730 read 0x02ad80 thru 0x02b0dd (0x35e bytes) 000495[1] 1788528973.578544 read 0x5f1890 thru 0x5f373f (0x1eb0 bytes) 000498[0] 1788528973.604229 write 0x3d6df9 thru 0x3d88b5 (0x1abd bytes) 000499[1] 1788528973.650775 trunc from 0x7b2631 to 0x3c72e1 (0x3eb34f bytes) All operations completed A-OK! PASS 16b (39s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 09:36:20 (1788528980) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 09:36:25 (1788528985) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.380332 s, 22.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.344963 s, 24.3 MB/s ldlm.namespaces.MGC192.168.204.115@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8c5dc6121000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8c5dd1e9c800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8c5dd1e9c800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8c5dd1e9c800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.9939 s, 7.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.92296 s, 26.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.76105 s, 38.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.01582 s, 17.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2221 s, 10.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.15389 s, 25.2 MB/s PASS 16d (55s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 09:37:20 (1788529040) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.185527 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779321 s, 13.5 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 09:37:28 (1788529048) TIMEOUT PASS 16f (30s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 09:37:58 (1788529078) TIMEOUT PASS 16g (31s) == sanityn test 16h: mmap read after truncate file ======= 09:38:29 (1788529109) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 09:38:37 (1788529117) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 09:38:45 (1788529125) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.23622 s, 21.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.76879 s, 14.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.49618 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.43931 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.2396 s, 8.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.68171 s, 9.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.27392 s, 8.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.52147 s, 17.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.97356 s, 13.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.46778 s, 7.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.62885 s, 7.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.28272 s, 11.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.41527 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.85556 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.04994 s, 8.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.56548 s, 10.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.92144 s, 9.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.37186 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.62307 s, 7.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (42s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 09:39:27 (1788529167) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Sep 4 09:40:18 EDT 2026 + fsx 2 done: rc=0 Fri Sep 4 09:40:18 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Fri Sep 4 09:40:19 EDT 2026 All operations completed A-OK! + fsx 4 done: rc=0 Fri Sep 4 09:40:19 EDT 2026 + fsx 5 done: rc=0 Fri Sep 4 09:40:19 EDT 2026 PASS 16k (66s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 09:40:34 (1788529234) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 09:40:47 (1788529247) mmap test1: basic mmap operation (PASS, 0.11814s) mmap test2: MAP_PRIVATE not write back (PASS, 0.096745s) mmap test3: concurrent mmap ops on two nodes (PASS, 15.852s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6346s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.25695s) mmap test6: check mmap write/read content on two nodes (PASS, 0.24859s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2015s) mmap test11: multi-thread shared mmap (PASS, 1.0891s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1306s) PASS 18 (33s) == sanityn test 19: test concurrent uncached read races ========================================================================= 09:41:20 (1788529280) oleg415-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 09:41:25 (1788529285) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 09:41:33 (1788529293) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 09:41:41 (1788529301) now is 1788529304 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 new atime is 1788529365 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 09:42:52 (1788529372) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118067 403 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (9s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 09:43:01 (1788529381) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 09:43:08 (1788529388) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 09:43:15 (1788529395) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 09:43:18 (1788529398) utime: good mknod times 1788529399 <= 1788529400 <= 1788529400 for /mnt/lustre/f26a utime: good mknod times 1788529399 <= 1788529400 <= 1788529400 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 09:43:26 (1788529406) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 09:43:34 (1788529414) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-04 09:43:36.000000000 -0400 Birth: 2026-09-04 09:43:36.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-04 09:43:36.000000000 -0400 Birth: 2026-09-04 09:43:36.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 09:43:43 (1788529423) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 09:43:52 (1788529432) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 09:44:00 (1788529440) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg415-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.115@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 09:44:21 (1788529461) -rw-r--r-- 1 root root 0 Sep 4 09:44 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 09:44:31 (1788529471) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43156 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 09:44:39 (1788529479) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 09:44:46 (1788529486) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 09:44:50 (1788529490) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.732 s, 30.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35237 s, 38.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.50887 s, 20.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.7996 s, 18.7 MB/s contention_seconds 4, contention_events 18 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.92853 s, 17.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.03233 s, 17.3 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05261 s, 49.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.0675 s, 49.1 MB/s PASS 32c (31s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 09:45:22 (1788529522) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 09:45:25 (1788529525) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 09:45:29 (1788529529) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 09:45:32 (1788529532) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 09:45:36 (1788529536) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c5dc6121000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c5dc6121000.ost_server_uuid in FULL state after 0 sec oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c5dc6121000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c5dc6121000.ost_server_uuid in IDLE state after 0 sec oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c5dc6121000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c5dc6121000.ost_server_uuid in IDLE state after 0 sec oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c5dc6121000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c5dc6121000.ost_server_uuid in IDLE state after 0 sec oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c5dc6121000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c5dc6121000.ost_server_uuid in IDLE state after 0 sec oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c5dc6121000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c5dc6121000.ost_server_uuid in IDLE state after 0 sec PASS 34 (147s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 09:48:03 (1788529683) fail_loc=0x80000317 Race attempt 0 Wait for 50463 50488 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50463 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.115@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre 192.168.204.115@tcp:/lustre 7.2G 39M 7.2G 1% /mnt/lustre2 - open/close 604 (time 1788529700.16 total 10.01 last 60.32) - open/close 1195 (time 1788529710.16 total 20.01 last 59.10) - open/close 1790 (time 1788529720.16 total 30.02 last 59.47) - open/close 2539 (time 1788529730.18 total 40.03 last 74.80) - open/close 3152 (time 1788529740.19 total 50.05 last 61.20) - open/close 3787 (time 1788529750.21 total 60.07 last 63.38) total: 4000 open/close in 64.12 seconds: 62.38 ops/second PASS 35 (73s) == sanityn test 36: handle ESTALE/open-unlink correctly == 09:49:16 (1788529756) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg415-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.978494 s, 53.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7498752) after_dd(7431168) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.32818 s, 39.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7498752) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.734137 s, 71.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7491584) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.02644 s, 51.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7491584) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.915723 s, 57.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7492608) after_dd(7438336) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.00053 s, 52.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7498752) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.702909 s, 74.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7491584) after_dd(7433216) after(7484416) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.0642 s, 49.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7484416) after_dd(7441408) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.771227 s, 68.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(8) *** before(7498752) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.890294 s, 58.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(9) *** before(7494656) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.882641 s, 59.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(10) *** before(7488512) after_dd(7441408) after(7492608) PASS 36 (307s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 09:54:23 (1788530063) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 - create 1230 (time 1788530077.68 total 10.00 last 122.94) - create 2397 (time 1788530087.69 total 20.01 last 116.65) - create 3520 (time 1788530097.69 total 30.01 last 112.26) - create 4365 (time 1788530107.69 total 40.01 last 84.50) - create 5480 (time 1788530117.70 total 50.02 last 111.40) - create 6688 (time 1788530127.71 total 60.03 last 120.71) - create 7874 (time 1788530137.71 total 70.03 last 118.59) - create 9076 (time 1788530147.71 total 80.04 last 120.14) total: 10000 create in 87.99 seconds: 113.65 ops/second PASS 37 (100s) == sanityn test 39a: file mtime does not change after rename ========================================================== 09:56:03 (1788530163) repeat after cancel_lru_locks PASS 39a (9s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 09:56:12 (1788530172) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 09:56:21 (1788530181) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 09:56:29 (1788530189) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 09:56:38 (1788530198) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 09:56:55 (1788530215) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (20s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 09:57:15 (1788530235) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (17s) == sanityn test 40e: pdirops: rename and others ======================================================================== 09:57:32 (1788530252) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 09:57:48 (1788530268) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 09:58:01 (1788530281) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 09:58:15 (1788530295) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (15s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 09:58:30 (1788530310) Conflict PASS 41d (17s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 09:58:47 (1788530327) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 09:59:02 (1788530342) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 09:59:18 (1788530358) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 09:59:32 (1788530372) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 09:59:46 (1788530386) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1154s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 10:19:00 (1788531540) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (18s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 10:19:18 (1788531558) open: File exists Conflict PASS 42b (16s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 10:19:34 (1788531574) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 10:19:49 (1788531589) Conflict PASS 42d (16s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 10:20:05 (1788531605) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (16s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 10:20:22 (1788531622) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 10:20:36 (1788531636) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 10:20:50 (1788531650) Conflict PASS 42h (16s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 10:21:06 (1788531666) PASS 43a (82s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 10:22:28 (1788531748) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 10:22:42 (1788531762) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 10:22:56 (1788531776) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 10:23:11 (1788531791) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 10:23:25 (1788531805) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 10:23:38 (1788531818) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 10:23:52 (1788531832) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 10:24:07 (1788531847) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 10:24:11 (1788531851) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (137s) == sanityn test 43k: unlink vs create ==================== 10:26:28 (1788531988) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1274s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 10:47:42 (1788533262) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 4 10:47:51 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 10:47:56 (1788533276) open: File exists Conflict PASS 44b (16s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 10:48:12 (1788533292) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (16s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 10:48:28 (1788533308) Conflict PASS 44d (18s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 10:48:46 (1788533326) Conflict PASS 44e (16s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 10:49:02 (1788533342) Conflict PASS 44f (16s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 10:49:18 (1788533358) Conflict PASS 44g (18s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 10:49:36 (1788533376) Conflict PASS 44h (17s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 10:49:53 (1788533393) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 10:49:58 (1788533398) PASS 45a (126s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 10:52:04 (1788533524) Conflict PASS 45b (17s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 10:52:21 (1788533541) Conflict PASS 45c (15s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 10:52:37 (1788533557) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (17s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 10:52:53 (1788533573) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 10:53:07 (1788533587) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 10:53:21 (1788533601) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (13s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 10:53:34 (1788533614) Conflict PASS 45h (15s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 10:53:49 (1788533629) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 10:53:54 (1788533634) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1328s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 11:16:02 (1788534962) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (18s) == sanityn test 46b: pdirops: link vs create ======================================================================== 11:16:20 (1788534980) open: File exists Conflict PASS 46b (15s) == sanityn test 46c: pdirops: link vs link ======================================================================== 11:16:35 (1788534995) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (17s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 11:16:52 (1788535012) Conflict PASS 46d (17s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 11:17:09 (1788535029) Conflict PASS 46e (17s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 11:17:27 (1788535047) Conflict PASS 46f (16s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 11:17:43 (1788535063) Conflict PASS 46g (16s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 11:17:59 (1788535079) Conflict PASS 46h (15s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 11:18:15 (1788535095) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 11:18:18 (1788535098) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 11:18:22 (1788535102) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 11:18:26 (1788535106) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (5s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 11:18:31 (1788535111) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 11:18:35 (1788535115) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 11:18:39 (1788535119) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 11:18:43 (1788535123) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (5s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 11:18:48 (1788535128) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.271916 s, 37.7 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (15s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 11:19:03 (1788535143) 0+1 records in 0+1 records out 158 bytes copied, 0.0322829 s, 4.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218470) - No such process PASS 51a (13s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 11:19:16 (1788535156) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0437633 s, 23.4 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25451 s, 4.1 MB/s 3145728 PASS 51b (23s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 11:19:39 (1788535179) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0551311 s, 18.6 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 11:19:53 (1788535193) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326895 s, 32.1 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 11:20:02 (1788535202) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 read lease(1) has applied. PASS 51e (11s) == sanityn test 54: rename locking ======================= 11:20:14 (1788535214) llite.lustre-ffff8c5dc6121000.xattr_cache=1 llite.lustre-ffff8c5dd1e9c800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 11:20:16.000000000 -0400 Modify: 2026-09-04 11:20:16.000000000 -0400 Change: 2026-09-04 11:20:16.000000000 -0400 Birth: 2026-09-04 11:20:17.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 11:20:23.000000000 -0400 Modify: 2026-09-04 11:20:24.000000000 -0400 Change: 2026-09-04 11:20:24.000000000 -0400 Birth: 2026-09-04 11:20:23.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 11:20:30.000000000 -0400 Modify: 2026-09-04 11:20:30.000000000 -0400 Change: 2026-09-04 11:20:30.000000000 -0400 Birth: 2026-09-04 11:20:30.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 11:20:30.000000000 -0400 Modify: 2026-09-04 11:20:30.000000000 -0400 Change: 2026-09-04 11:20:30.000000000 -0400 Birth: 2026-09-04 11:20:30.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 11:20:36.000000000 -0400 Modify: 2026-09-04 11:20:38.000000000 -0400 Change: 2026-09-04 11:20:38.000000000 -0400 Birth: 2026-09-04 11:20:36.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 11:20:36.000000000 -0400 Modify: 2026-09-04 11:20:38.000000000 -0400 Change: 2026-09-04 11:20:38.000000000 -0400 Birth: 2026-09-04 11:20:36.000000000 -0400 PASS 54 (37s) == sanityn test 55a: rename vs unlink target dir ========= 11:20:51 (1788535251) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (15s) == sanityn test 55b: rename vs unlink source dir ========= 11:21:07 (1788535267) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (15s) == sanityn test 55c: rename vs unlink orphan target dir == 11:21:22 (1788535282) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (21s) == sanityn test 55d: rename file vs link ================= 11:21:43 (1788535303) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (18s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 11:22:01 (1788535321) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (5s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 11:22:06 (1788535326) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (23s) == sanityn test 55g: rename: race with trylock =========== 11:22:29 (1788535349) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (27s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 11:22:57 (1788535377) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788535379 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.45894 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (144s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 11:25:20 (1788535520) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788535530 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.00061 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (143s) == sanityn test 60: Verify data_version behaviour ======== 11:27:43 (1788535663) PASS 60 (11s) == sanityn test 70a: cd directory && rm directory ======== 11:27:54 (1788535674) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 11:28:05 (1788535685) PASS 70b (11s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 11:28:16 (1788535696) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0119807 s, 5.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (5s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 11:28:21 (1788535701) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0409314 s, 1.6 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (5s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 11:28:27 (1788535707) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 11:28:31 (1788535711) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0202168 s, 3.2 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 11:28:35 (1788535715) llite.lustre-ffff8c5dc6121000.xattr_cache=1 llite.lustre-ffff8c5dd1e9c800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (10s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 11:28:45 (1788535725) llite.lustre-ffff8c5dc6121000.xattr_cache=1 llite.lustre-ffff8c5dd1e9c800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 11:28:55 (1788535735) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0433146 s, 23.6 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0260075 s, 39.4 kB/s 231311: get lock2 231312: get lock1 231311: done 231311: get lock1 231312: done 231312: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231312: cannot get lock2: Resource deadlock avoided 231312: exit rc=0 231311: done 231311: exit rc=0 PASS 74 (13s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 11:29:07 (1788535747) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0459515 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581617 s, 18.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231893 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 11:29:20 (1788535760) opening files: ........ closing files: ........ PASS 76 (67s) == sanityn test 77a: check FIFO NRS policy =============== 11:30:27 (1788535827) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.233253 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261226 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619688 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549784 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116381 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100461 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126608 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103515 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11879 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140431 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106074 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141618 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118883 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981902 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477664 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137988 s, 7.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439175 s, 23.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699873 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759411 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711728 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036117 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518363 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054119 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892132 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542817 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618366 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634064 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103312 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645733 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663493 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676668 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764471 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076538 s, 13.7 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (10s) == sanityn test 77b: check CRR-N NRS policy ============== 11:30:38 (1788535838) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.384187 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743815 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099775 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12966 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213744 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22013 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22251 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145429 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09086 s, 11.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128209 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149318 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193735 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146599 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123781 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192779 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164937 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242286 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563311 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301965 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883172 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083343 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463105 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735371 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100045 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135334 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107184 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083424 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599521 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647396 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670723 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07453 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07494 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795017 s, 13.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.417241 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140417 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529143 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750072 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998895 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148055 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133041 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12564 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190743 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18739 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153599 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144035 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748187 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196311 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226393 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440887 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154324 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164697 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812608 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127243 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143812 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109286 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130758 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795499 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155101 s, 67.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994273 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150826 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14831 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624037 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229013 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107634 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225543 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153047 s, 6.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (16s) == sanityn test 77c: check ORR NRS policy ================ 11:30:54 (1788535854) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.31782 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546892 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08235 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136756 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674788 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117338 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127677 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170201 s, 6.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148202 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119682 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10883 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142772 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201522 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124672 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184594 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249019 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288259 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722937 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02286 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111302 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114883 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105914 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0072947 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129365 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865647 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419276 s, 250 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100284 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996695 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130324 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15465 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177245 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112778 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164921 s, 63.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.315623 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222652 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045147 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800549 s, 13.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746503 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518703 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104615 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818935 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297951 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934153 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118258 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126322 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110776 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866383 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15415 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161177 s, 6.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185182 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496252 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490749 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642786 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569843 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642019 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618904 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529714 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815269 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629294 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505365 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676113 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709602 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679014 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562818 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788402 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121096 s, 8.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.304126 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834266 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835084 s, 12.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10905 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165471 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218902 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211099 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216071 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229179 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189524 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257454 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232265 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234857 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247975 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28676 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297717 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328743 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158646 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126366 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138086 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154097 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182806 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444171 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152447 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078066 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692673 s, 15.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182007 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169885 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893344 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205557 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187177 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21408 s, 4.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115899 s, 9.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (16s) == sanityn test 77d: check TRR nrs policy ================ 11:31:10 (1788535870) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.198323 s, 84.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503273 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775322 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547657 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116135 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120586 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164147 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139462 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127342 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107953 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146272 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200383 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178481 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128053 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187503 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220314 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219635 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564698 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498657 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520013 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095478 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608506 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061535 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513193 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569015 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537994 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059381 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767093 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633389 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648445 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068356 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129381 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074582 s, 14.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246544 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365663 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476151 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379753 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053351 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100305 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109443 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101419 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13002 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111534 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124474 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158137 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133543 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131516 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164773 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164332 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20706 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763852 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758156 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769349 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00374587 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759374 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140653 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144026 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640402 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328852 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234059 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599537 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155165 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733202 s, 14.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156955 s, 6.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181375 s, 57.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522828 s, 20.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.276742 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035194 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504449 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955307 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125773 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06154 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459152 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965722 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275643 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115953 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996563 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109211 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1812 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850291 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229672 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236884 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158371 s, 6.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022736 s, 46.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374347 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516853 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342338 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585566 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580584 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101603 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558989 s, 18.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722046 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952844 s, 11.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060011 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619175 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864526 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725806 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750917 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099582 s, 10.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (13s) == sanityn test 77e: check TBF NID nrs policy ============ 11:31:24 (1788535884) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.15@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.309618 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227996 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435114 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163043 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691587 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038674 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352203 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375275 s, 27.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189975 s, 5.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121624 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165979 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209892 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12957 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224541 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222755 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239271 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289543 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817158 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648374 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432734 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845306 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383235 s, 27.4 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392649 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263535 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509342 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569622 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435124 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578288 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083186 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827475 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405612 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610575 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00817264 s, 128 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.212257 s, 79.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606579 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468007 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13331 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137356 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162372 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11576 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170839 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197494 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180584 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144513 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191303 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154197 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161038 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148454 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137531 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173735 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525125 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506485 s, 20.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045169 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274543 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202499 s, 51.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821048 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570814 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310859 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787971 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862038 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855155 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068614 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402537 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498902 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00567297 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00712343 s, 147 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.34101 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627584 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148415 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737149 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915179 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209876 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217602 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419122 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585531 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297516 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174013 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19805 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486351 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200802 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326611 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179141 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401625 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387491 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031749 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247748 s, 423 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273633 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109208 s, 9.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112202 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114068 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106509 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350444 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651321 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222111 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141613 s, 74.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156759 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134808 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348575 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097747 s, 10.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255902 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330575 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508041 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972477 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549734 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122969 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104522 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106101 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13627 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164041 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131801 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125571 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162104 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149865 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109368 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19822 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141926 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00946435 s, 111 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977674 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00584352 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11313 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137105 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166332 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017598 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100149 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863321 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118426 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082809 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664825 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777664 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614661 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530345 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460816 s, 22.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (25s) == sanityn test 77f: check TBF JobID nrs policy ========== 11:31:49 (1788535909) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.273327 s, 61.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362917 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629983 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771235 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100851 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157947 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133114 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150274 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105368 s, 10.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109263 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168208 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169974 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202126 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12424 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22991 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235036 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246035 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152406 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205525 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180032 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153012 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14382 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126384 s, 83.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00501606 s, 209 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119597 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110338 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241252 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102648 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965415 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774185 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704469 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544779 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440102 s, 23.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.207368 s, 80.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548043 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708004 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861474 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826241 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143362 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165153 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104391 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180342 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15004 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24127 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219062 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173782 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209524 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227926 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222548 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216073 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276881 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199798 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375395 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744458 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466716 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569952 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415411 s, 25.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491751 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584792 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580405 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593776 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374229 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607033 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622469 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689611 s, 15.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687914 s, 15.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.223478 s, 75.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341535 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342756 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421028 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937279 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131481 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114256 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684864 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976615 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837495 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155733 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111147 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135142 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157779 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165018 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154887 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197177 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278688 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00761257 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643738 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020378 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848512 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726842 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0069944 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739762 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814357 s, 12.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855379 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119357 s, 87.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100603 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00642952 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00537311 s, 195 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872413 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095386 s, 11.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.293296 s, 57.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264189 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320184 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024178 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316745 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383202 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101275 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456042 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337382 s, 31.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556207 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195555 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14445 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11688 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15261 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995378 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119225 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131239 s, 8.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00734961 s, 143 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288046 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817142 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237822 s, 44.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357069 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321627 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00471226 s, 223 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103366 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111959 s, 93.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018393 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973092 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132553 s, 79.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00875922 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00484153 s, 217 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102698 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00827194 s, 127 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (23s) == sanityn test 77g: Change TBF type directly ============ 11:32:12 (1788535932) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (13s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 11:32:25 (1788535945) oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg415-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg415-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg415-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg415-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 PASS 77h (14s) == sanityn test 77i: Change rank of TBF rule ============= 11:32:39 (1788535959) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (27s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 11:33:06 (1788535986) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.445628 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040851 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577011 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309347 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101284 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116323 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914638 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107843 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122557 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157844 s, 6.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142161 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151643 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155927 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106253 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117308 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123388 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119322 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316251 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204008 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195855 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070166 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786109 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631801 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907542 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859884 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351762 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781813 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414206 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433297 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458767 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482722 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535773 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673343 s, 15.6 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.77081 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6442 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (70s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 11:34:16 (1788536056) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg415-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg415-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg415-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg415-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26803 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262491 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062003 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552531 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323363 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104228 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290007 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881856 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433889 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496315 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157467 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186671 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177898 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195289 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220621 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176715 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207279 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041367 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528441 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499571 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384634 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056815 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683719 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608683 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675838 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491234 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707483 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692028 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081074 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102514 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105893 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102915 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111792 s, 9.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2925 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7795 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg415-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg415-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg415-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg415-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30547 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989612 s, 10.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00811611 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111566 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984424 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128477 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228274 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141954 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261255 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188064 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176953 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214523 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300062 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224094 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268055 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353726 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312986 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894829 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624613 s, 16.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588801 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890988 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105196 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111192 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872602 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692666 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862693 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628566 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824371 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133047 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10328 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154111 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148989 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120797 s, 8.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2626 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8494 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (165s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 11:37:01 (1788536221) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg415-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg415-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg415-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg415-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26801 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727954 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732367 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929348 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755906 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102294 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121319 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146307 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158154 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160219 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168814 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161315 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144394 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186379 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180833 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198045 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236223 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705255 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785513 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745236 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938862 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876679 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925473 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869387 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111267 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114913 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09069 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120077 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118658 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918164 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112511 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152962 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148572 s, 7.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9894 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0218 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg415-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg415-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg415-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg415-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.36024 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046284 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602049 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656849 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537481 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100468 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136393 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177989 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847551 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198018 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145347 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232416 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141452 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130755 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210896 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282565 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329993 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350253 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093253 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104614 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101622 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100275 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113624 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710241 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794638 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740849 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092394 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993567 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861661 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111849 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118445 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141316 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152778 s, 6.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9579 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8783 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (165s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 11:39:46 (1788536386) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.603786 s, 27.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304378 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075234 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858486 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112037 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13863 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922957 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150777 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178219 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184903 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262529 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266605 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292402 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276581 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310691 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289318 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393996 s, 2.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684816 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677016 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506623 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430293 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652785 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725881 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576407 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645169 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12653 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932415 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942694 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112401 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114978 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817849 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965566 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100273 s, 10.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.1482 s, 8.6 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0332 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.15@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.15@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.614108 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078773 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102315 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1638 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976326 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236048 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185223 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235748 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22048 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180043 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204247 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256185 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352606 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399754 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318025 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34106 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396897 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc6121000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120026 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137782 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044594 s, 23.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105426 s, 99.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577332 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13262 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131345 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203012 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18788 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240077 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174298 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289113 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240303 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257566 s, 4.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198637 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249359 s, 4.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6527 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93874 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.15@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.60808 s, 27.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437353 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464155 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477563 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316558 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490463 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156686 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230371 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179238 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253506 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257233 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288215 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287492 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304318 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295811 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304406 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352869 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946032 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579871 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565375 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100833 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393858 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729196 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669539 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791805 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119521 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815399 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706521 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830935 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101032 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899526 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790465 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114524 s, 9.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84966 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.64477 s, 18.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.591304 s, 28.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366873 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592807 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128497 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148569 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112455 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159119 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113651 s, 9.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11333 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127767 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126108 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149658 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187209 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12888 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142108 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24361 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220724 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635295 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541184 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608933 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119403 s, 8.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823087 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054887 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948626 s, 11.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858362 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102312 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077192 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104904 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870916 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109025 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108543 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100832 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08744 s, 12.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7905 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97457 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1471 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94384 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.05486 s, 15.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541547 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090707 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14799 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621541 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14394 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212653 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122075 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180932 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126016 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898244 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23954 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271707 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201115 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719224 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111625 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24473 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606395 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753102 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586071 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974436 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885364 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701958 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125605 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712838 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797856 s, 13.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806228 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091115 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115174 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998468 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104759 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10564 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117083 s, 9.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8882 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7456 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.421732 s, 39.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339348 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909109 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896705 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269322 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273925 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202482 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252572 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246637 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274225 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201653 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325646 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38116 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355731 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252585 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293747 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417905 s, 2.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc6121000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105381 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125495 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164652 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12659 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114161 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159179 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194031 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153932 s, 6.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136066 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13783 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171248 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178482 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181474 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160351 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174538 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185523 s, 5.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7759 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93759 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.286814 s, 58.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717194 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479596 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110223 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707975 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174227 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069236 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211011 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206553 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176426 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142958 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254782 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289447 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282232 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306176 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227659 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302122 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264254 s, 39.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00737742 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661281 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00904634 s, 116 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799973 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338465 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0091161 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435268 s, 24.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652672 s, 16.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906762 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776639 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00751148 s, 140 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221864 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115258 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122809 s, 85.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162677 s, 64.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6053 s, 9.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95092 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3864 s, 8.5 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95914 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (480s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 11:47:47 (1788536867) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.378326 s, 44.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527368 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768429 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232601 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25441 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209522 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264153 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262817 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246019 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302409 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293495 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362907 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364947 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331642 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36904 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390021 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392263 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633085 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974466 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808752 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095497 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059904 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839243 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08585 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873358 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122688 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116862 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124203 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115076 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166988 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120833 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121313 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185143 s, 5.7 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.759 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91339 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (57s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 11:48:44 (1788536924) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.15@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.15@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.46004 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297261 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217414 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563513 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799857 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993275 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892953 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14161 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157077 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223102 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191726 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111633 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13854 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216782 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173084 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238977 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272877 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560749 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638388 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458916 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580676 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730151 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663454 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139654 s, 75.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988558 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102898 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772379 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12067 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955592 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113901 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116049 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130582 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187818 s, 5.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.64351 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0123 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (59s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 11:49:43 (1788536983) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.15@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.715485 s, 23.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938924 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101392 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127821 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16037 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168155 s, 6.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181339 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146301 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142507 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217307 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14669 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175156 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189355 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137419 s, 7.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144521 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177657 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266228 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846907 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714355 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934988 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470189 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11323 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896374 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626819 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652567 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125529 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14071 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700437 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749317 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843833 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882419 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895765 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920182 s, 11.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.58963 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.46763 s, 19.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (52s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 11:50:36 (1788537036) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.542213 s, 30.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428902 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755459 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477701 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707607 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680916 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653102 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127439 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120741 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167578 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19166 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175695 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210824 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241804 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253992 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254703 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243568 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656008 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107687 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101366 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601362 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744343 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066599 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799963 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122674 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073068 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912556 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909152 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103484 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107597 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103655 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107048 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123605 s, 8.5 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0889 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9219 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.60913 s, 12.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92291 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (108s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 11:52:23 (1788537143) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.01899 s, 16.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333886 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662405 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308374 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140432 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991348 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431349 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976354 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126777 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725651 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114196 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213299 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158993 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153523 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103372 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234682 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2793 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132111 s, 79.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574342 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579002 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00720861 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038402 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457232 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865494 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614975 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670812 s, 15.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287265 s, 36.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571823 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613594 s, 17.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157031 s, 66.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00741282 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00851107 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319185 s, 32.9 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9193 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0083 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (82s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 11:53:45 (1788537225) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.50291 s, 33.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327501 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685167 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459763 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487601 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710889 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11119 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925493 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228003 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264435 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15506 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167208 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284403 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192396 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220328 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260718 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305196 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644082 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779128 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686528 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890586 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121364 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925398 s, 11.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885671 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086405 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087691 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125031 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918325 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138604 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961814 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986233 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995996 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102824 s, 10.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.43437 s, 11.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9268 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg415-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.63921 s, 26.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261218 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258659 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914924 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105966 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927727 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103754 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118688 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141018 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144975 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118602 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12305 s, 8.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166024 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153841 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148028 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139581 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17606 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055647 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174766 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306985 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045853 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259879 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329727 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0028324 s, 370 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321184 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106729 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927291 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101857 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839965 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633492 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867326 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847063 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288473 s, 36.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.2163 s, 6.9 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9523 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0569 s, 8.7 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0087 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (159s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 11:56:24 (1788537384) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg415-client.virtnet /mnt/lustre (opts:) Stopping client oleg415-client.virtnet /mnt/lustre opts: Stopping clients: oleg415-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg415-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg415-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg415-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.15,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg415-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.15,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg415-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.15,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69495/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.15,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg415-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing load_modules_local oleg415-server: kmemleak disabled oleg415-server: Loading modules from /home/green/git/lustre-release/lustre oleg415-server: detected 4 online CPUs by sysfs oleg415-server: MODOPTS_LIBCFS= oleg415-server: Force libcfs to create 2 CPU partitions oleg415-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre Starting client oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre Started clients oleg415-client.virtnet: 192.168.204.115@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 Starting client oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 Started clients oleg415-client.virtnet: 192.168.204.115@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c5dc5ba8800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c5dd1e0c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c5dc5ba8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c5dd1e0c800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.448401 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742626 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170762 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962102 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223823 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146307 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220887 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225802 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230109 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855016 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294911 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278538 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425105 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235011 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363778 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455191 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342742 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc5ba8800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2949 s, 3.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266715 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310036 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272423 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280689 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282089 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328822 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc5ba8800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306915 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308559 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297508 s, 3.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297304 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339525 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288772 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289992 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308198 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300841 s, 3.5 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1343 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.28009 s, 19.9 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.614634 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268775 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508993 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375347 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048202 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525492 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067202 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193597 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216851 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242051 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224173 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284575 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296742 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240658 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318899 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309122 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324352 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc5ba8800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602699 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260611 s, 40.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590422 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285083 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309386 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00484894 s, 216 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00685703 s, 153 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112955 s, 92.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388755 s, 27.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427796 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427997 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675557 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543762 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109966 s, 95.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034903 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509183 s, 20.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2942 s, 10.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90213 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (214s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 11:59:59 (1788537599) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg415-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 12:00:17 (1788537617) On MGS 192.168.204.115, active = nodemap.active=1 On MGS 192.168.204.115, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.80071 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639762 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05069 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154696 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171032 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14672 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689235 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256035 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167058 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168636 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299199 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276695 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305143 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc5ba8800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170983 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094818 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173571 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215038 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452223 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46301 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488502 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477437 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484559 s, 2.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481406 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494896 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.508779 s, 2.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521602 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.513666 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523779 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.528043 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.542296 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.572222 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.551824 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.55374 s, 1.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.61016 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20068 s, 20.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.115, TBF.id = On MGS 192.168.204.115, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.579502 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748065 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088681 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866274 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138159 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204424 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212409 s, 4.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143718 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197858 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248694 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259548 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199881 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220909 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247576 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280032 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281717 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25887 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc5ba8800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc5ba8800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540773 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478538 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060346 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577745 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525807 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647873 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873751 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964522 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643611 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636692 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803923 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407162 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522545 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055001 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011405 s, 91.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774295 s, 13.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg415-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.93161 s, 11.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.98816 s, 35.1 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg415-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg415-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 On MGS 192.168.204.115, active = nodemap.active=1 On MGS 192.168.204.115, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.827065 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979211 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149765 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167019 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28818 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31426 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323168 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324699 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318402 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287573 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322031 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328701 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344977 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397805 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290283 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396636 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc5ba8800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc5ba8800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298954 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177696 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20024 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182196 s, 5.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208437 s, 5.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17646 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192753 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218943 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228806 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221628 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21796 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235699 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237673 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231776 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226893 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229731 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254003 s, 4.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73511 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93426 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.115, TBF.id = On MGS 192.168.204.115, active = nodemap.active=0 PASS 77kj (181s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 12:03:18 (1788537798) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 12:03:29 (1788537809) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg415-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.47238 s, 302 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg415-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0706 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (56s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 12:04:26 (1788537866) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.671924 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872709 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112853 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125521 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28011 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192775 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253662 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33744 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362529 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319561 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420507 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398522 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10641 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253492 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479295 s, 2.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc5ba8800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416039 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460387 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778797 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085357 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133773 s, 7.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10237 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937252 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10899 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113828 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867669 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864218 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152563 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981183 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890864 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114816 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130066 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159199 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159383 s, 6.6 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8554 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.11186 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.735922 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797455 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715664 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133131 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173403 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143675 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1951 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13259 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228363 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13805 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172069 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308725 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103971 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521168 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136199 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141598 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17435 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c5dc5ba8800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c5dc5ba8800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463912 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475161 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468549 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.516096 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.467465 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487289 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.488733 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.499942 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495262 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495541 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.498251 s, 2.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52375 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.517167 s, 2.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.513751 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.513687 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.522188 s, 2.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg415-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg415-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1689 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.36008 s, 19.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (100s) == sanityn test 77o: Changing rank should not panic ====== 12:06:07 (1788537967) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 12:06:19 (1788537979) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (119s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 12:08:19 (1788538099) mds.MDS.mdt.nrs_policies=tbf oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg415-client: oleg415-server: ssh exited with exit code 36 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg415-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg415-client: oleg415-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (41s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 12:08:59 (1788538139) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273319 kill md_thread xargs PASS 77r (58s) == sanityn test 77s: Check TBF LRU shrinker ============== 12:09:57 (1788538197) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (68s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 12:11:05 (1788538265) oleg415-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg415-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg415-client: oleg415-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 12:11:14 (1788538274) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935647 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 12:11:16.000000000 -0400 Modify: 2026-09-04 12:11:16.000000000 -0400 Change: 2026-09-04 12:11:16.000000000 -0400 Birth: 2026-09-04 12:11:17.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 12:11:33 (1788538293) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 12:11:37 (1788538297) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 12:11:40 (1788538300) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 12:11:44 (1788538304) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 12:11:48 (1788538308) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 12:11:51 (1788538311) - open/close 635 (time 1788538327.67 total 10.01 last 63.43) - open/close 1248 (time 1788538337.68 total 20.03 last 61.20) - open/close 1848 (time 1788538347.69 total 30.03 last 59.99) - open/close 2410 (time 1788538357.69 total 40.04 last 56.16) total: 2500 open/close in 41.59 seconds: 60.10 ops/second snapshot_time 1788538462.783242648 secs.nsecs start_time 1788538315.182010651 secs.nsecs elapsed_time 147.601231997 secs.nsecs open 2505 samples [usecs] 141 26962 5144731 20328617713 close 2504 samples [usecs] 183 12592 1219789 1603971711 mknod 2501 samples [usecs] 576 26823 4480285 16198556857 unlink 2500 samples [usecs] 718 87263 6338024 46309250988 mkdir 2 samples [usecs] 2787 4106 6893 24626605 rmdir 2 samples [usecs] 9136 14779 23915 301885337 rename 2500 samples [usecs] 491 79108 19646772 319206766748 getattr 3760 samples [usecs] 9 9841 340660 516763546 parallel_rename_file 2500 samples [usecs] 491 79108 19646772 319206766748 crossdir_rename 2500 samples [usecs] 491 79108 19646772 319206766748 readpage 18 samples [usecs] 4724 45906 373641 10433353379 PASS 81d (157s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 12:14:28 (1788538468) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7614 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7614 PASS 82 (9s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 12:14:37 (1788538477) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (5s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 12:14:42 (1788538482) fail_loc=0x8000060b PASS 84 (15s) == sanityn test 85: Lustre API root cache race =========== 12:14:57 (1788538497) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788538499 Finishing test test1 at 1788538505 PASS 85 (14s) == sanityn test 90: open/create and unlink striped directory ========================================================== 12:15:11 (1788538511) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 12:15:15 (1788538515) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 12:15:19 (1788538519) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 12:15:23 (1788538523) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5779 0x1693 0x240000400 1 5740 0x166c 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5741 0x166d 0x280000400 0 5780 0x1694 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 12:15:40 (1788538540) osc.lustre-OST0000-osc-ffff8c5dc5ba8800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8c5dd1e0c800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8c5dc5ba8800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8c5dd1e0c800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.351366 s, 23.3 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.92378 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 289905 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0919787 s, 44.5 kB/s osc.lustre-OST0000-osc-ffff8c5dc5ba8800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c5dd1e0c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c5dc5ba8800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c5dd1e0c800.idle_timeout=debug PASS 94 (18s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 12:15:58 (1788538558) llite.lustre-ffff8c5dc5ba8800.fast_read=0 llite.lustre-ffff8c5dd1e0c800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0668356 s, 31.4 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8c5dc5ba8800.fast_read=1 llite.lustre-ffff8c5dd1e0c800.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 12:16:19 (1788538579) llite.lustre-ffff8c5dc5ba8800.fast_read=0 llite.lustre-ffff8c5dd1e0c800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00901445 s, 1.4 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8c5dc5ba8800.fast_read=1 llite.lustre-ffff8c5dd1e0c800.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 12:16:34 (1788538594) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 12:16:38 (1788538598) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0122409 s, 335 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (11s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 12:16:49 (1788538609) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.340289 s, 6.2 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (10s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 12:16:59 (1788538619) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.205629 s, 10.2 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (10s) == sanityn test 100e: DoM: read on open and file size ==== 12:17:10 (1788538630) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00678155 s, 1.8 MB/s PASS 100e (11s) == sanityn test 101a: Discard DoM data on unlink ========= 12:17:20 (1788538640) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00973177 s, 421 kB/s PASS 101a (7s) == sanityn test 101b: Discard DoM data on rename ========= 12:17:27 (1788538647) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.017385 s, 236 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 12:17:35 (1788538655) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0145896 s, 281 kB/s PASS 101c (10s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 12:17:46 (1788538666) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 40 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf40:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 12:17:54 (1788538674) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788538678 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788538681 Finishing test test23 at 1788538682 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 12:18:09 (1788538689) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 12:18:12 (1788538692) fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 12:18:40 (1788538720) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 12:18:45 (1788538725) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0216047 s, 47.4 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938183 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 12:18:47.000000000 -0400 Modify: 2026-09-04 12:18:47.000000000 -0400 Change: 2026-09-04 12:18:47.000000000 -0400 Birth: 2026-09-04 12:18:47.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938183 -rw-r--r-- 1 0 0 1788538727 1788538727 1788538727 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938183 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 12:18:47.000000000 -0400 Modify: 2026-09-04 12:18:47.000000000 -0400 Change: 2026-09-04 12:18:47.000000000 -0400 Birth: 2026-09-04 12:18:47.000000000 -0400 1788538727 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 12:18:53 (1788538733) PASS 106c (9s) == sanityn test 107a: Basic grouplock conflict =========== 12:19:02 (1788538742) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.3167 s, 33.1 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5748 0x1674 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 First grouplock blocks second one PASS 107a (11s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 12:19:13 (1788538753) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.270144 s, 38.8 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5789 0x169d 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 12:19:27 (1788538767) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0071552 s, 1.4 MB/s llite.lustre-ffff8c5dc5ba8800.hybrid_io=0 llite.lustre-ffff8c5dd1e0c800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.6318 s, 1.8 MB/s PASS 108a (11s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 12:19:39 (1788538779) 192.168.204.115@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg415-client.virtnet /mnt/lustre (opts:) 192.168.204.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg415-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 192.168.204.115@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg415-client.virtnet /mnt/lustre (opts:) Stopping client oleg415-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing load_modules_local oleg415-server: kmemleak disabled oleg415-server: Loading modules from /home/green/git/lustre-release/lustre oleg415-server: detected 4 online CPUs by sysfs oleg415-server: MODOPTS_LIBCFS= oleg415-server: Force libcfs to create 2 CPU partitions oleg415-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 192.168.204.115@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.115@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg415-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg415-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing load_modules_local oleg415-server: kmemleak disabled oleg415-server: Loading modules from /home/green/git/lustre-release/lustre oleg415-server: detected 4 online CPUs by sysfs oleg415-server: MODOPTS_LIBCFS= oleg415-server: Force libcfs to create 2 CPU partitions oleg415-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre Starting client: oleg415-client.virtnet: -o user_xattr,flock 192.168.204.115@tcp:/lustre /mnt/lustre2 PASS 109 (72s) == sanityn test 110: do not grant another lock on resend ========================================================== 12:20:52 (1788538852) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-04 12:20:53.000000000 -0400 Modify: 2026-09-04 12:20:53.000000000 -0400 Change: 2026-09-04 12:20:53.000000000 -0400 Birth: 2026-09-04 12:20:54.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg415-client.virtnet oleg415-client.virtnet PASS 110 (84s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 12:22:16 (1788538936) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 12:22:20 (1788538940) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 12:22:23 (1788538943) lustre-OST0000-osc-ffff8c5dc5ba8000 active. lustre-OST0001-osc-ffff8c5dc5ba8000 active. lustre-OST0000-osc-ffff8c5dc5ba8000 active. lustre-OST0001-osc-ffff8c5dc5ba8000 active. MGC192.168.204.115@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 12:22:32 (1788538952) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (5s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 12:22:37 (1788538957) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 12:22:41 (1788538961) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 12:22:45 (1788538965) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0395021 s, 518 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.158684 s, 129 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 12:22:54 (1788538974) PASS 122 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 12:23:05 (1788538985) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2018 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.234 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2776 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2132 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2632 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.235 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3046 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.12 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1746 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2737 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3029 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3308 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3465 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3096 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2675 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2869 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2882 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4157 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg415-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c5dc5ba8000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c5dc5ba8000.ost_server_uuid in FULL state after 0 sec PASS 200 (86s) cleanup: ====================================================== == sanityn test complete, duration 10643 sec ============= 12:24:33 (1788539073) === sanityn: start cleanup 12:24:35 (1788539075) === Stopping clients: oleg415-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg415-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 12:29:42 (1788539382) ===