-----============= acceptance-small: sanityn ============----- Mon Sep 7 13:13:22 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg653-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 13:13:40 (1788801220) === Starting client oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 Mount client oleg653-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 Started clients oleg653-client.virtnet: 192.168.206.153@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg653-client.virtnet: executing check_config_client /mnt/lustre oleg653-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg653-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff942d82f2e800.idle_timeout=debug osc.lustre-OST0000-osc-ffff942d8601d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff942d82f2e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff942d8601d800.idle_timeout=debug disable quota as required oleg653-server: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 13:14:09 (1788801249) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378271 s, 27.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7627] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 13:14:11 (1788801251) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 13:14:18 (1788801258) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 13:14:24 (1788801264) -rw-r--r-- 1 root root 0 Sep 7 13:14 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 13:14:30 (1788801270) -rw-r--r-- 1 root root 0 Sep 7 13:14 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 13:14:36 (1788801276) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (5s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 13:14:41 (1788801281) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 13:14:48 (1788801288) total 1 -rwxrwxrwx 1 root root 0 Sep 7 13:14 f2a -rwxrwxrwx 1 root root 0 Sep 7 13:14 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 7 13:14 f2a -rwxrwxrwx 1 root root 0 Sep 7 13:14 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0113108 s, 45.3 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 13:14:54 (1788801294) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 13:14:57 (1788801297) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.304173 s, 6.9 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 13:15:03 (1788801303) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 13:15:09 (1788801309) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 13:15:15 (1788801315) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 13:15:21 (1788801321) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 13:15:27 (1788801327) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 13:15:33 (1788801333) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (5s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 13:15:39 (1788801339) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 13:15:45 (1788801345) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564067 s, 18.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0968549 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 13:15:52 (1788801352) 1+0 records in 1+0 records out 1 byte copied, 0.00388518 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0113724 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00858811 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0103636 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0176967 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0213796 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0145974 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0262432 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0150763 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00908433 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0182789 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.017849 s, 0.1 kB/s PASS 10a (6s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 13:15:58 (1788801358) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0250164 s, 123 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0371421 s, 110 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000739687 s, 4.2 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 13:16:04 (1788801364) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 13:16:10 (1788801370) warning: '-runtime' deprecated, use '-t runtime' instead - link 382 (time 1788801382.95 total 10.01 last 38.15) - link 752 (time 1788801392.95 total 20.02 last 36.97) - link 1146 (time 1788801402.95 total 30.02 last 39.40) - link 1539 (time 1788801412.98 total 40.05 last 39.21) - link 1940 (time 1788801423.01 total 50.08 last 39.98) - link 2322 (time 1788801433.01 total 60.08 last 38.19) - link 2675 (time 1788801443.02 total 70.08 last 35.28) - link 3080 (time 1788801453.03 total 80.09 last 40.46) - link 3467 (time 1788801463.03 total 90.10 last 38.69) total: 3806 link in 100.02 seconds: 38.05 ops/second using seed 3117873325 running for 100 seconds - stat 4756 (time 1788801384 ; total 11 ; last 11) - stat 10000 (time 1788801393 ; total 20 ; last 9) - stat 16052 (time 1788801404 ; total 31 ; last 11) - stat 20000 (time 1788801410 ; total 37 ; last 6) - stat 25947 (time 1788801421 ; total 48 ; last 11) - stat 29424 (time 1788801432 ; total 59 ; last 11) - stat 30000 (time 1788801434 ; total 61 ; last 2) - stat 32645 (time 1788801445 ; total 72 ; last 11) - stat 35687 (time 1788801456 ; total 83 ; last 11) - stat 38343 (time 1788801467 ; total 94 ; last 11) total: 39546 stats in 100 seconds: 395.459991 stats/second - unlinked 0 (time 1788801423 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3806) error: No such file or directory total: 3805 unlinks in 92 seconds: 41.358696 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21429) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21431) - No such process PASS 12 (151s) == sanityn test 13: test directory page revocation ======= 13:18:41 (1788801521) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 13:18:50 (1788801530) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 22617 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 13:18:57 (1788801537) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 23197 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 13:19:05 (1788801545) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23775 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 13:19:13 (1788801553) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24357 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 13:19:20 (1788801560) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24939 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 13:19:28 (1788801568) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (52s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 13:20:20 (1788801620) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.422671 s, 19.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.439527 s, 19.1 MB/s Chance of close/open is 1 in 50 Seed set to 1625 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read skipping zero size read skipping zero length punch hole skipping zero length punch hole skipping zero length punch hole skipping zero size read 000010[1] 1788801626.262043 write 0x811f78 thru 0x815bca (0x3c53 bytes) 000011[0] 1788801626.267013 write 0x57363b thru 0x5783be (0x4d84 bytes) 000012[1] 1788801626.307157 write 0x95d9d6 thru 0x96ae71 (0xd49c bytes) 000014[1] 1788801626.334514 punch from 0x5724d4 to 0x57f5c6 (0xd0f3 bytes) truncating to largest ever: 0x9d7d91 000015[1] 1788801626.415428 trunc from 0x96ae72 to 0x9d7d90 (0x6cf1f bytes) 000016[1] 1788801626.460507 read 0x0d9112 thru 0x0dc7e0 (0x36cf bytes) 000017[0] 1788801626.497208 mapwrite 0x3771ad thru 0x37a619 (0x346d bytes) 000018[1] 1788801626.623539 read 0x1654b8 thru 0x174eea (0xfa33 bytes) 000019[1] 1788801626.662516 mapwrite 0x1d374a thru 0x1d3b90 (0x447 bytes) 000021[1] 1788801626.795281 trunc from 0x9d7d91 to 0x4af528 (0x528868 bytes) 000024[0] 1788801626.830224 mapread 0x4a114f thru 0x4af528 (0xe3da bytes) 000027[1] 1788801626.967504 trunc from 0x4af529 to 0x345046 (0x16a4e2 bytes) 000028[1] 1788801627.006999 write 0x3b3b81 thru 0x3b5b37 (0x1fb7 bytes) 000030[1] 1788801627.062271 punch from 0x2f161c to 0x2f84dd (0x6ec2 bytes) 000031[1] 1788801627.082287 trunc from 0x3b5b38 to 0x230a43 (0x1850f4 bytes) 000034[0] 1788801627.142703 read 0x0b24f0 thru 0x0bd7de (0xb2ef bytes) 000035[0] 1788801627.205084 write 0x9042f9 thru 0x909392 (0x509a bytes) 000036[0] 1788801627.248333 write 0x8ebef9 thru 0x8f6d69 (0xae71 bytes) 000038[0] 1788801627.293290 write 0x1f3764 thru 0x1fc679 (0x8f16 bytes) 000039[0] 1788801627.323650 read 0x87cbb8 thru 0x87ef0e (0x2357 bytes) 000041[1] 1788801627.364379 write 0x1867d8 thru 0x1914ce (0xacf7 bytes) 000042[1] 1788801627.399298 punch from 0x8c415b to 0x8ca8a4 (0x674a bytes) 000043[1] 1788801627.424591 mapwrite 0x765ec7 thru 0x773e35 (0xdf6f bytes) 000044[1] 1788801627.646183 punch from 0x05b033 to 0x063a2f (0x89fd bytes) 000045[1] 1788801627.662419 write 0x74d41a thru 0x74e614 (0x11fb bytes) 000046[0] 1788801627.697554 trunc from 0x909393 to 0x30104b (0x608347 bytes) 000049[1] 1788801627.763809 punch from 0x034b15 to 0x03613c (0x1628 bytes) 000051[1] 1788801627.810895 mapread 0x19d63b thru 0x1a8a3b (0xb401 bytes) 000053[1] 1788801627.881406 read 0x08b4bb thru 0x095ace (0xa614 bytes) 000055[1] 1788801627.987362 trunc from 0x30104c to 0x85f514 (0x55e4c9 bytes) 000057[1] 1788801628.080655 mapwrite 0x40ccab thru 0x411922 (0x4c78 bytes) 000058[1] 1788801628.223417 mapread 0x2d5d70 thru 0x2e11c9 (0xb45a bytes) 000060[1] 1788801628.283414 mapwrite 0x7e5d28 thru 0x7e894f (0x2c28 bytes) 000061[1] 1788801628.387465 write 0x78ca1f thru 0x7940a8 (0x768a bytes) 000062[0] 1788801628.388559 trunc from 0x85f515 to 0x6eb729 (0x173deb bytes) 000064[0] 1788801628.517171 punch from 0x20376d to 0x20dfd4 (0xa868 bytes) 000066[1] 1788801628.561334 read 0x0d299c thru 0x0d2e0a (0x46f bytes) 000067[0] 1788801628.585019 mapwrite 0x3a9293 thru 0x3b1070 (0x7dde bytes) 000068[1] 1788801628.712779 trunc from 0x6eb72a to 0x6d1ea3 (0x19886 bytes) 000070[0] 1788801628.803364 punch from 0x543f9d to 0x54ce16 (0x8e7a bytes) 000071[0] 1788801628.828523 punch from 0x3bd140 to 0x3cc7c4 (0xf685 bytes) 000077[1] 1788801628.914089 trunc from 0x6d1ea4 to 0x048e78 (0x68902b bytes) 000078[1] 1788801628.982480 mapwrite 0x2f60e1 thru 0x2fdad0 (0x79f0 bytes) 000079[0] 1788801629.078958 mapread 0x12bd21 thru 0x12e8ea (0x2bca bytes) 000081[1] 1788801629.153202 write 0x31f637 thru 0x32387a (0x4244 bytes) 000083[1] 1788801629.168621 read 0x0e9bdd thru 0x0f74db (0xd8ff bytes) 000085[1] 1788801629.240218 punch from 0x2e1c9e to 0x2f0436 (0xe799 bytes) 000086[1] 1788801629.247531 mapwrite 0x6f5e9c thru 0x7000a4 (0xa209 bytes) 000087[0] 1788801629.463259 mapwrite 0x78eef3 thru 0x79cf2a (0xe038 bytes) 000088[1] 1788801629.749767 punch from 0x61b22d to 0x61f235 (0x4009 bytes) 000091[1] 1788801629.905718 read 0x50e1e3 thru 0x5141d3 (0x5ff1 bytes) 000092[1] 1788801629.948452 mapread 0x6a372d thru 0x6aea22 (0xb2f6 bytes) 000093[1] 1788801630.025173 read 0x554ec7 thru 0x55c0ba (0x71f4 bytes) 000094[0] 1788801630.056738 punch from 0x311090 to 0x316077 (0x4fe8 bytes) 000095[0] 1788801630.088056 read 0x3c2352 thru 0x3cf414 (0xd0c3 bytes) 000096[0] 1788801630.114223 read 0x2a2034 thru 0x2ab385 (0x9352 bytes) 000097[0] 1788801630.192803 mapwrite 0x57a732 thru 0x581614 (0x6ee3 bytes) 000099[0] 1788801630.336781 mapread 0x0c9b9d thru 0x0caa2b (0xe8f bytes) 000100[1] 1788801630.359659 punch from 0x511ebe to 0x5160e1 (0x4224 bytes) 000101[0] 1788801630.402684 write 0x0ee0ef thru 0x0f9638 (0xb54a bytes) 000103[0] 1788801630.436135 write 0x4cb129 thru 0x4da122 (0xeffa bytes) 000105[1] 1788801630.499576 trunc from 0x79cf2b to 0x443b11 (0x359419 bytes) 000110[0] 1788801630.766240 punch from 0x0c0dfe to 0x0ce9c3 (0xdbc6 bytes) 000111[0] 1788801630.827121 punch from 0x34e4db to 0x358e4c (0xa972 bytes) 000112[0] 1788801630.863593 read 0x1d3219 thru 0x1d598a (0x2772 bytes) 000113[1] 1788801630.902244 punch from 0x2c0432 to 0x2c39b8 (0x3587 bytes) 000117[1] 1788801631.007808 mapwrite 0x9b5ac2 thru 0x9b8321 (0x2860 bytes) 000120[1] 1788801631.178868 mapwrite 0x711bc9 thru 0x71dffc (0xc434 bytes) 000121[0] 1788801631.274086 mapread 0x92ad0f thru 0x9353e0 (0xa6d2 bytes) 000122[1] 1788801631.340326 mapwrite 0x1d9f6a thru 0x1dd353 (0x33ea bytes) 000123[1] 1788801631.473663 write 0x031aa0 thru 0x03a02c (0x858d bytes) 000125[0] 1788801631.534670 punch from 0x7f6bf3 to 0x7fdbea (0x6ff8 bytes) 000126[1] 1788801631.565494 mapread 0x1c4a7e thru 0x1cb126 (0x66a9 bytes) 000127[0] 1788801631.595298 mapread 0x852cb0 thru 0x856c12 (0x3f63 bytes) 000128[1] 1788801631.653235 write 0x043a24 thru 0x04914a (0x5727 bytes) 000129[1] 1788801631.706014 read 0x0c7857 thru 0x0cd44e (0x5bf8 bytes) 000130[1] 1788801631.736257 mapwrite 0x07e30c thru 0x080f36 (0x2c2b bytes) 000131[0] 1788801631.825868 read 0x5b9f97 thru 0x5bf6c4 (0x572e bytes) 000133[0] 1788801631.879517 trunc from 0x9b8322 to 0x94c3e4 (0x6bf3d bytes) 000135[0] 1788801631.913094 read 0x83dcb6 thru 0x8452ab (0x75f6 bytes) 000136[0] 1788801631.992774 mapread 0x535057 thru 0x543ead (0xee57 bytes) 000138[1] 1788801632.143306 write 0x125547 thru 0x1308f3 (0xb3ad bytes) 000142[1] 1788801632.270727 write 0x4fab35 thru 0x508918 (0xdde4 bytes) 000143[0] 1788801632.296309 punch from 0x367862 to 0x371e75 (0xa614 bytes) 000144[1] 1788801632.352807 mapread 0x2b314f thru 0x2b9bd4 (0x6a86 bytes) 000145[1] 1788801632.424292 trunc from 0x94c3e5 to 0x7ec8af (0x15fb35 bytes) 000148[0] 1788801632.461797 mapwrite 0x8e996b thru 0x8f19ee (0x8084 bytes) 000150[1] 1788801632.620191 trunc from 0x8f19ef to 0x75fcff (0x191cef bytes) 000152[1] 1788801632.734131 write 0x1214b0 thru 0x126e5f (0x59b0 bytes) 000154[0] 1788801632.784604 mapread 0x2fe8a9 thru 0x3046bc (0x5e14 bytes) 000157[0] 1788801632.871212 mapread 0x248e2e thru 0x2539b6 (0xab89 bytes) 000158[0] 1788801632.903681 read 0x6b6b81 thru 0x6c3ce4 (0xd164 bytes) 000159[1] 1788801633.039804 punch from 0x1785d4 to 0x17becd (0x38fa bytes) 000160[1] 1788801633.088526 read 0x16ccb5 thru 0x177b4f (0xae9b bytes) 000162[0] 1788801633.100621 write 0x6f355d thru 0x6f43b8 (0xe5c bytes) 000165[1] 1788801633.226519 write 0x4065a4 thru 0x408e11 (0x286e bytes) 000167[1] 1788801633.337343 mapread 0x24b24e thru 0x24bf22 (0xcd5 bytes) 000169[0] 1788801633.414400 write 0x12361c thru 0x131577 (0xdf5c bytes) 000171[0] 1788801633.485308 write 0x56e17a thru 0x57cac1 (0xe948 bytes) 000172[0] 1788801633.509803 mapread 0x732db7 thru 0x73c3f0 (0x963a bytes) 000176[0] 1788801633.622785 punch from 0x19ec71 to 0x1abfb9 (0xd349 bytes) 000177[1] 1788801633.673896 write 0x99e508 thru 0x9a0a35 (0x252e bytes) 000178[0] 1788801633.708458 read 0x79d9f3 thru 0x7a7ee1 (0xa4ef bytes) 000180[1] 1788801633.923642 read 0x627434 thru 0x6367ed (0xf3ba bytes) 000181[0] 1788801633.970963 mapwrite 0x62d2bd thru 0x630cda (0x3a1e bytes) 000182[0] 1788801634.150850 trunc from 0x9a0a36 to 0x3ddc7d (0x5c2db8 bytes) 000184[1] 1788801634.261660 read 0x10047b thru 0x10260d (0x2193 bytes) 000185[1] 1788801634.349882 mapwrite 0x92f8c4 thru 0x931e99 (0x25d6 bytes) 000186[0] 1788801634.539921 write 0x96ad22 thru 0x973000 (0x82df bytes) 000187[1] 1788801634.583186 mapwrite 0x64830c thru 0x65469f (0xc394 bytes) 000188[1] 1788801634.678656 read 0x271dce thru 0x27ea3d (0xcc70 bytes) 000189[1] 1788801634.769893 read 0x171bf7 thru 0x17667e (0x4a88 bytes) 000190[0] 1788801634.812608 trunc from 0x973001 to 0x0debc7 (0x894439 bytes) 000191[0] 1788801634.875202 mapread 0x03a114 thru 0x049c6a (0xfb57 bytes) 000193[1] 1788801634.952582 trunc from 0x0debc8 to 0x9a2c8c (0x8c40c5 bytes) 000194[1] 1788801635.008659 write 0x69ce5a thru 0x6ab5f8 (0xe79f bytes) 000196[0] 1788801635.075291 punch from 0x399dcb to 0x39b53a (0x1770 bytes) 000198[1] 1788801635.119769 mapread 0x1f6f6e thru 0x202724 (0xb7b7 bytes) 000199[0] 1788801635.170794 mapread 0x47a85d thru 0x481c4c (0x73f0 bytes) 000200[0] 1788801635.215601 read 0x909e9a thru 0x912e9f (0x9006 bytes) 000201[1] 1788801635.224050 mapread 0x78dc65 thru 0x7976c6 (0x9a62 bytes) 000203[1] 1788801635.249064 trunc from 0x9a2c8d to 0x5a599a (0x3fd2f2 bytes) 000208[0] 1788801635.337587 read 0x2b00ee thru 0x2bb92f (0xb842 bytes) 000209[1] 1788801635.404741 write 0x015914 thru 0x01d742 (0x7e2f bytes) 000210[0] 1788801635.443800 trunc from 0x5a599b to 0x2134fd (0x39249d bytes) 000211[0] 1788801635.530758 mapwrite 0x9be1d8 thru 0x9c048c (0x22b5 bytes) 000212[1] 1788801635.760624 trunc from 0x9c048d to 0x150464 (0x870028 bytes) 000215[0] 1788801635.851773 trunc from 0x150465 to 0x7328ff (0x5e249b bytes) 000216[0] 1788801635.926724 read 0x15189c thru 0x15693a (0x509f bytes) 000218[0] 1788801636.049451 mapread 0x48217d thru 0x490569 (0xe3ed bytes) 000219[0] 1788801636.088607 mapwrite 0x7e0666 thru 0x7eae36 (0xa7d1 bytes) 000220[1] 1788801636.200609 mapread 0x2ff90d thru 0x30e934 (0xf028 bytes) 000221[1] 1788801636.250569 write 0x8244bf thru 0x8295e9 (0x512b bytes) 000223[1] 1788801636.347844 trunc from 0x8295ea to 0x73fed7 (0xe9712 bytes) 000225[1] 1788801636.409175 write 0x03bd8c thru 0x049375 (0xd5ea bytes) 000226[1] 1788801636.421450 punch from 0x37a515 to 0x37a5ca (0xb6 bytes) 000229[1] 1788801636.479747 trunc from 0x73fed8 to 0x365ec1 (0x3da016 bytes) 000232[1] 1788801636.583043 trunc from 0x365ec2 to 0x2142ec (0x151bd5 bytes) 000236[0] 1788801636.700303 trunc from 0x2142ed to 0x016703 (0x1fdbe9 bytes) 000238[1] 1788801636.877845 trunc from 0x016704 to 0x38e58c (0x377e89 bytes) 000240[1] 1788801636.967104 mapwrite 0x5b41f3 thru 0x5b9348 (0x5156 bytes) 000241[0] 1788801637.158051 mapwrite 0x5f95e8 thru 0x601201 (0x7c1a bytes) 000242[0] 1788801637.394867 read 0x1e4b38 thru 0x1f165c (0xcb25 bytes) 000243[0] 1788801637.425770 read 0x3f865d thru 0x3fdeab (0x584f bytes) truncating to largest ever: 0x9eab2f 000244[1] 1788801637.484961 trunc from 0x601202 to 0x9eab2e (0x3e992d bytes) 000245[1] 1788801637.604849 mapread 0x171c4f thru 0x17f340 (0xd6f2 bytes) 000246[0] 1788801637.680834 mapwrite 0x2f9486 thru 0x2fe27b (0x4df6 bytes) 000247[0] 1788801637.935721 mapread 0x664c80 thru 0x66eb05 (0x9e86 bytes) 000250[1] 1788801638.051994 trunc from 0x9eab2f to 0x5e63bd (0x404771 bytes) 000255[0] 1788801638.196738 punch from 0x3ec1e2 to 0x3f9e83 (0xdca2 bytes) 000258[1] 1788801638.274610 punch from 0x274024 to 0x27e1c9 (0xa1a6 bytes) 000259[0] 1788801638.300721 mapread 0x4c7e80 thru 0x4d5cfd (0xde7e bytes) 000260[0] 1788801638.410576 trunc from 0x5e63be to 0x71f735 (0x139378 bytes) 000262[1] 1788801638.540537 mapwrite 0x3d0569 thru 0x3d91b4 (0x8c4c bytes) 000264[1] 1788801638.868700 trunc from 0x71f736 to 0x285fd2 (0x499763 bytes) 000265[1] 1788801638.997207 mapwrite 0x989c1e thru 0x98bc81 (0x2064 bytes) 000266[1] 1788801639.226974 write 0x442c43 thru 0x44b1dc (0x859a bytes) 000267[0] 1788801639.268814 mapwrite 0x855af3 thru 0x85eea3 (0x93b1 bytes) 000270[1] 1788801639.611668 trunc from 0x98bc82 to 0x269cb0 (0x721fd1 bytes) 000271[1] 1788801639.726696 punch from 0x149f23 to 0x14c51f (0x25fd bytes) 000272[0] 1788801639.737910 mapwrite 0x4da1e1 thru 0x4e083f (0x665f bytes) 000273[1] 1788801639.952562 punch from 0x28b534 to 0x297929 (0xc3f6 bytes) 000274[1] 1788801640.030779 write 0x34eb15 thru 0x354b46 (0x6032 bytes) 000275[1] 1788801640.078823 trunc from 0x4e0840 to 0x726db2 (0x246573 bytes) 000276[0] 1788801640.194297 mapwrite 0x506d26 thru 0x511511 (0xa7ec bytes) 000277[1] 1788801640.357867 punch from 0x203fd6 to 0x20a83f (0x686a bytes) 000278[1] 1788801640.399431 write 0x353b76 thru 0x355f53 (0x23de bytes) 000281[1] 1788801640.524637 mapwrite 0x5e0f21 thru 0x5e39fc (0x2adc bytes) 000282[0] 1788801640.842498 mapread 0x4de93a thru 0x4e38c0 (0x4f87 bytes) 000283[0] 1788801640.910630 write 0x46d1f8 thru 0x46d8f2 (0x6fb bytes) 000285[0] 1788801640.998648 mapwrite 0x751472 thru 0x75c3ce (0xaf5d bytes) 000286[0] 1788801641.224619 write 0x263c4a thru 0x2739e3 (0xfd9a bytes) 000287[0] 1788801641.272036 trunc from 0x75c3cf to 0x96e35e (0x211f90 bytes) 000288[0] 1788801641.285851 punch from 0x2c3df7 to 0x2cf69d (0xb8a7 bytes) 000289[1] 1788801641.319861 mapread 0x5a75d0 thru 0x5aac29 (0x365a bytes) 000290[1] 1788801641.368283 punch from 0x47c826 to 0x480e98 (0x4673 bytes) 000291[1] 1788801641.430298 trunc from 0x96e35f to 0x95e81e (0xfb40 bytes) 000292[0] 1788801641.482494 trunc from 0x95e81f to 0x113417 (0x84b407 bytes) 000293[0] 1788801641.507513 mapread 0x0c2c9c thru 0x0ce3e9 (0xb74e bytes) 000294[0] 1788801641.560620 punch from 0x0015b4 to 0x009399 (0x7de6 bytes) 000295[1] 1788801641.570069 write 0x4e7daf thru 0x4f6b21 (0xed73 bytes) 000296[0] 1788801641.616405 read 0x3a01d9 thru 0x3a0e27 (0xc4f bytes) 000297[0] 1788801641.679404 trunc from 0x4f6b22 to 0x19cd40 (0x359de1 bytes) 000298[1] 1788801641.736406 trunc from 0x19cd41 to 0x8f9151 (0x75c411 bytes) 000299[0] 1788801641.796221 trunc from 0x8f9152 to 0x4ed52c (0x40bc25 bytes) 000301[0] 1788801641.860057 write 0x8d07cf thru 0x8de8d2 (0xe104 bytes) 000302[1] 1788801641.882726 write 0x5241a7 thru 0x52fc44 (0xba9e bytes) 000303[1] 1788801641.921728 mapwrite 0x5455e2 thru 0x5473b8 (0x1dd7 bytes) 000305[1] 1788801642.127237 read 0x33da62 thru 0x342fa6 (0x5545 bytes) 000306[1] 1788801642.183456 punch from 0x225b33 to 0x22deaf (0x837d bytes) 000307[1] 1788801642.227480 mapread 0x36a007 thru 0x371d60 (0x7d5a bytes) 000308[1] 1788801642.298412 read 0x20d58e thru 0x21a9c0 (0xd433 bytes) 000309[0] 1788801642.311549 mapwrite 0x40a6ae thru 0x413553 (0x8ea6 bytes) 000311[0] 1788801642.536043 write 0x0c0e91 thru 0x0ca3b8 (0x9528 bytes) 000312[0] 1788801642.761851 read 0x5b0c8a thru 0x5b5848 (0x4bbf bytes) 000313[1] 1788801642.817496 punch from 0x7cc359 to 0x7cf48c (0x3134 bytes) 000315[1] 1788801642.911009 read 0x4da2f2 thru 0x4e67a6 (0xc4b5 bytes) 000319[0] 1788801643.043733 write 0x9629ae thru 0x967ac1 (0x5114 bytes) 000320[1] 1788801643.087958 mapwrite 0x49ca88 thru 0x4a1cd7 (0x5250 bytes) 000321[1] 1788801643.251398 punch from 0x668064 to 0x66b051 (0x2fee bytes) 000323[0] 1788801643.327640 write 0x82ddc3 thru 0x8320fc (0x433a bytes) 000324[1] 1788801643.379191 write 0x413809 thru 0x419626 (0x5e1e bytes) 000325[0] 1788801643.433264 mapread 0x23b90a thru 0x249c05 (0xe2fc bytes) 000326[0] 1788801643.457597 mapread 0x378c7f thru 0x37a9c9 (0x1d4b bytes) 000327[0] 1788801643.478265 read 0x7acf97 thru 0x7b718a (0xa1f4 bytes) 000328[1] 1788801643.589577 mapwrite 0x6186c9 thru 0x6279ae (0xf2e6 bytes) 000329[1] 1788801644.307404 punch from 0x4a4575 to 0x4ad52d (0x8fb9 bytes) 000330[1] 1788801644.339825 mapread 0x23f601 thru 0x244509 (0x4f09 bytes) 000331[0] 1788801644.419021 write 0x4c375a thru 0x4c9667 (0x5f0e bytes) 000335[0] 1788801644.541525 mapread 0x2ca230 thru 0x2d4d8d (0xab5e bytes) 000336[0] 1788801644.642451 mapread 0x4763db thru 0x476df6 (0xa1c bytes) 000339[1] 1788801644.678240 trunc from 0x967ac2 to 0x04c32f (0x91b792 bytes) 000341[0] 1788801644.876916 punch from 0x0136a4 to 0x019bed (0x654a bytes) 000342[0] 1788801644.930457 write 0x68f186 thru 0x69e39c (0xf217 bytes) 000343[0] 1788801644.979416 trunc from 0x69e39d to 0x8a146a (0x2030ce bytes) 000344[1] 1788801645.103804 trunc from 0x8a146b to 0x9dd5d6 (0x13c16c bytes) 000345[1] 1788801645.222832 trunc from 0x9dd5d7 to 0x8d3df8 (0x1097de bytes) 000347[0] 1788801645.285742 mapread 0x62a331 thru 0x63414e (0x9e1e bytes) 000348[1] 1788801645.400711 mapwrite 0x8e086b thru 0x8e7abb (0x7251 bytes) 000350[1] 1788801645.596996 mapwrite 0x6f8a68 thru 0x6fe0b7 (0x5650 bytes) 000351[0] 1788801645.772198 punch from 0x32fe9e to 0x3393b1 (0x9514 bytes) 000353[0] 1788801645.900097 punch from 0x677d78 to 0x680a91 (0x8d1a bytes) 000354[1] 1788801646.063418 trunc from 0x8e7abc to 0x8bb142 (0x2c979 bytes) 000355[1] 1788801646.260460 punch from 0x3f8047 to 0x3f96cd (0x1687 bytes) 000359[0] 1788801646.381583 punch from 0x03d39d to 0x047c97 (0xa8fb bytes) 000360[1] 1788801646.466266 trunc from 0x8bb143 to 0x22616f (0x694fd3 bytes) 000367[0] 1788801646.637115 write 0x34a395 thru 0x34dcae (0x391a bytes) 000369[0] 1788801646.748638 write 0x4a98b0 thru 0x4af6e1 (0x5e32 bytes) 000372[1] 1788801646.807190 read 0x374424 thru 0x37bd03 (0x78e0 bytes) 000375[1] 1788801646.987181 read 0x1b2474 thru 0x1c0fe6 (0xeb73 bytes) 000376[0] 1788801647.015624 mapwrite 0x35af8c thru 0x361e5c (0x6ed1 bytes) 000377[0] 1788801647.510097 read 0x30000f thru 0x307e23 (0x7e15 bytes) 000378[1] 1788801647.646370 punch from 0x2e6486 to 0x2f4813 (0xe38e bytes) 000380[1] 1788801647.804399 punch from 0x1e3372 to 0x1ef33a (0xbfc9 bytes) 000382[0] 1788801647.951370 mapwrite 0x109744 thru 0x1189e7 (0xf2a4 bytes) 000383[0] 1788801648.236485 read 0x0c002a thru 0x0c54a3 (0x547a bytes) 000385[1] 1788801648.332203 mapread 0x1a4ec1 thru 0x1b1c34 (0xcd74 bytes) 000386[0] 1788801648.521664 trunc from 0x4af6e2 to 0x67e55c (0x1cee7b bytes) 000387[0] 1788801648.614434 mapread 0x59371e thru 0x59d4c9 (0x9dac bytes) 000388[1] 1788801648.691471 read 0x643a4b thru 0x6537eb (0xfda1 bytes) 000391[0] 1788801648.844266 mapread 0x632c20 thru 0x633bea (0xfcb bytes) 000392[0] 1788801648.872662 mapread 0x03995d thru 0x03e01f (0x46c3 bytes) 000394[0] 1788801648.961635 mapwrite 0x04bb17 thru 0x05991f (0xde09 bytes) 000395[0] 1788801649.106158 mapwrite 0x4776fe thru 0x480e28 (0x972b bytes) 000396[0] 1788801649.433753 write 0x6ebb33 thru 0x6f6c92 (0xb160 bytes) 000398[1] 1788801649.491105 mapread 0x13278f thru 0x136a25 (0x4297 bytes) 000399[0] 1788801649.602220 trunc from 0x6f6c93 to 0x50c025 (0x1eac6d bytes) 000400[0] 1788801649.680630 mapread 0x0a4a82 thru 0x0b43fb (0xf97a bytes) 000402[0] 1788801649.746371 mapwrite 0x0bc614 thru 0x0c3273 (0x6c60 bytes) 000403[1] 1788801649.935857 read 0x0897e1 thru 0x08a41e (0xc3e bytes) 000405[1] 1788801649.972982 mapread 0x36ae3a thru 0x371fd8 (0x719f bytes) 000406[0] 1788801650.075577 write 0x1e95a7 thru 0x1eedf9 (0x5853 bytes) 000407[0] 1788801650.127773 read 0x12459d thru 0x126d91 (0x27f5 bytes) 000408[1] 1788801650.175413 read 0x173dd8 thru 0x17caea (0x8d13 bytes) 000410[1] 1788801650.244649 read 0x18163f thru 0x18c008 (0xa9ca bytes) 000415[1] 1788801650.418122 read 0x479fb3 thru 0x485cbf (0xbd0d bytes) 000416[0] 1788801650.476100 write 0x8902ee thru 0x892820 (0x2533 bytes) 000417[0] 1788801650.530126 mapwrite 0x96486e thru 0x96f804 (0xaf97 bytes) 000418[1] 1788801650.706201 trunc from 0x96f805 to 0x63284a (0x33cfba bytes) 000419[0] 1788801650.834221 write 0x624969 thru 0x6344d8 (0xfb70 bytes) 000420[1] 1788801650.895782 write 0x8f933a thru 0x90337f (0xa046 bytes) 000421[1] 1788801650.918601 read 0x53bc64 thru 0x549ee7 (0xe284 bytes) 000422[0] 1788801651.032095 write 0x19a4ae thru 0x1a0353 (0x5ea6 bytes) 000423[0] 1788801651.107711 read 0x4ed98a thru 0x4f4591 (0x6c08 bytes) 000424[1] 1788801651.146064 read 0x2f9521 thru 0x2f9857 (0x337 bytes) 000425[1] 1788801651.210593 mapwrite 0x105149 thru 0x1073c3 (0x227b bytes) 000427[1] 1788801651.451276 mapwrite 0x3ebd68 thru 0x3f9ae8 (0xdd81 bytes) 000428[1] 1788801651.675695 trunc from 0x903380 to 0x3120ae (0x5f12d1 bytes) 000429[0] 1788801651.795641 read 0x300c3e thru 0x310bb0 (0xff73 bytes) 000433[1] 1788801652.024572 trunc from 0x3120af to 0x011b58 (0x300556 bytes) 000437[0] 1788801652.123132 punch from 0x011a62 to 0x011b58 (0xf7 bytes) 000438[1] 1788801652.155221 trunc from 0x011b59 to 0x80653d (0x7f49e5 bytes) 000439[0] 1788801652.212773 write 0x267a09 thru 0x26c0e2 (0x46da bytes) 000441[0] 1788801652.277326 mapread 0x01f6e9 thru 0x0235e9 (0x3f01 bytes) 000442[0] 1788801652.315693 write 0x3dfa33 thru 0x3e851d (0x8aeb bytes) 000443[0] 1788801652.329578 mapread 0x10d90f thru 0x11baa1 (0xe193 bytes) 000444[0] 1788801652.420458 read 0x53ce5a thru 0x54664c (0x97f3 bytes) 000445[0] 1788801652.461179 mapwrite 0x1e7376 thru 0x1f3a75 (0xc700 bytes) 000447[0] 1788801652.559272 read 0x523b53 thru 0x525e69 (0x2317 bytes) 000450[0] 1788801652.593269 mapread 0x12d070 thru 0x131282 (0x4213 bytes) 000451[1] 1788801652.613087 mapwrite 0x50248d thru 0x50a25d (0x7dd1 bytes) 000454[0] 1788801652.897505 trunc from 0x80653e to 0x098420 (0x76e11d bytes) 000456[1] 1788801652.990110 mapread 0x021077 thru 0x028809 (0x7793 bytes) 000463[1] 1788801653.080702 mapwrite 0x6a658e thru 0x6b5aba (0xf52d bytes) 000464[1] 1788801653.289217 punch from 0x4865ee to 0x48e215 (0x7c28 bytes) 000465[1] 1788801653.303407 mapread 0x358f67 thru 0x35db63 (0x4bfd bytes) 000467[0] 1788801653.396562 punch from 0x45297f to 0x46084f (0xded1 bytes) 000468[1] 1788801653.427011 punch from 0x138740 to 0x144a1a (0xc2db bytes) 000470[0] 1788801653.470318 mapwrite 0x381e15 thru 0x38568e (0x387a bytes) 000471[0] 1788801653.586061 mapwrite 0x85e55a thru 0x868769 (0xa210 bytes) 000473[1] 1788801653.841968 punch from 0x5f5340 to 0x5fc224 (0x6ee5 bytes) 000475[0] 1788801653.921444 read 0x7e955a thru 0x7f8cf3 (0xf79a bytes) 000476[1] 1788801653.942550 punch from 0x62b8f6 to 0x62c6d4 (0xddf bytes) 000477[1] 1788801654.002358 write 0x129a9e thru 0x12e808 (0x4d6b bytes) 000478[1] 1788801654.043841 punch from 0x85f8e6 to 0x863730 (0x3e4b bytes) 000479[0] 1788801654.089867 punch from 0x0e373f to 0x0ed7e3 (0xa0a5 bytes) 000480[1] 1788801654.165006 mapwrite 0x78d047 thru 0x79497d (0x7937 bytes) 000483[0] 1788801654.352509 trunc from 0x86876a to 0x77ebe5 (0xe9b84 bytes) 000487[0] 1788801654.470557 trunc from 0x77ebe6 to 0x04883e (0x7363a7 bytes) 000489[1] 1788801654.519713 read 0x03782d thru 0x03b483 (0x3c57 bytes) 000491[1] 1788801654.544506 mapread 0x0484ea thru 0x04883e (0x355 bytes) 000492[1] 1788801654.575592 write 0x43b141 thru 0x43caa3 (0x1963 bytes) 000493[1] 1788801654.616570 trunc from 0x43caa4 to 0x61430d (0x1d786a bytes) 000495[1] 1788801654.729828 punch from 0x47d34d to 0x48c4e3 (0xf197 bytes) 000498[0] 1788801654.784339 punch from 0x131cc5 to 0x1385a7 (0x68e3 bytes) 000500[0] 1788801654.838748 punch from 0x1fec05 to 0x1ff121 (0x51d bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 1654 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000005[1] 1788801655.774857 write 0x034000 thru 0x037481 (0x3482 bytes) truncating to largest ever: 0x95ce31 000007[1] 1788801655.818180 trunc from 0x037482 to 0x95ce30 (0x9259af bytes) 000010[1] 1788801655.869784 trunc from 0x95ce31 to 0x1f0caf (0x76c181 bytes) 000012[1] 1788801655.937430 mapwrite 0x7c4000 thru 0x7c4eaa (0xeab bytes) 000015[0] 1788801656.128731 write_OD 0x96a000 thru 0x96afff (0x1000 bytes) 000016[0] 1788801656.183035 read_OD 0x06b000 thru 0x075fff (0xb000 bytes) 000017[0] 1788801656.205849 trunc from 0x96b000 to 0x3a07f7 (0x5ca808 bytes) 000018[0] 1788801656.260212 mapwrite 0x16d000 thru 0x1770f6 (0xa0f7 bytes) 000020[0] 1788801656.402909 mapwrite 0x157000 thru 0x1659b5 (0xe9b6 bytes) 000021[1] 1788801656.546974 trunc from 0x3a07f8 to 0x0d4d86 (0x2cba71 bytes) 000023[1] 1788801656.627673 write 0x277000 thru 0x27c559 (0x555a bytes) 000024[0] 1788801656.719722 trunc from 0x27c55a to 0x2c3aa3 (0x4754a bytes) 000026[1] 1788801656.833871 write 0x57f000 thru 0x58cbf6 (0xdbf7 bytes) 000028[1] 1788801656.907531 write 0x369000 thru 0x36c959 (0x395a bytes) 000029[0] 1788801656.912804 mapwrite 0x826000 thru 0x82f4e0 (0x94e1 bytes) 000031[1] 1788801657.249087 read 0x26f000 thru 0x273ebe (0x4ebf bytes) 000032[1] 1788801657.294902 mapwrite 0x26c000 thru 0x277592 (0xb593 bytes) 000034[0] 1788801657.573601 mapwrite 0x2e9000 thru 0x2f6f06 (0xdf07 bytes) 000035[1] 1788801657.827597 trunc from 0x82f4e1 to 0x17100a (0x6be4d6 bytes) 000037[0] 1788801657.922269 mapwrite 0x33d000 thru 0x3413db (0x43dc bytes) 000038[0] 1788801658.102020 write_OD 0x571000 thru 0x57bfff (0xb000 bytes) 000039[1] 1788801658.255089 read 0x1ed000 thru 0x1ed361 (0x362 bytes) 000041[0] 1788801658.461163 trunc from 0x57c000 to 0x2e1364 (0x29ac9b bytes) 000042[1] 1788801658.584043 punch from 0x2b5cf9 to 0x2bb3ab (0x56b3 bytes) 000043[1] 1788801658.654196 write 0x860000 thru 0x86e9c7 (0xe9c8 bytes) 000044[1] 1788801658.664689 trunc from 0x86e9c8 to 0x92eab6 (0xc00ef bytes) 000046[0] 1788801658.775602 read_OD 0x5d2000 thru 0x5d4fff (0x3000 bytes) 000047[1] 1788801658.903065 mapread 0x44f000 thru 0x45b40b (0xc40c bytes) 000050[1] 1788801659.106661 mapread 0x03d000 thru 0x040239 (0x323a bytes) 000051[1] 1788801659.148529 trunc from 0x92eab7 to 0x751a28 (0x1dd08e bytes) 000053[0] 1788801659.282708 read_OD 0x07b000 thru 0x07bfff (0x1000 bytes) 000054[1] 1788801659.334094 write 0x560000 thru 0x560eb0 (0xeb1 bytes) 000056[0] 1788801659.382204 punch from 0x2f4966 to 0x2f708d (0x2728 bytes) 000057[1] 1788801659.484035 mapread 0x229000 thru 0x237ef9 (0xeefa bytes) 000058[0] 1788801659.636649 read_OD 0x2aa000 thru 0x2b0fff (0x7000 bytes) 000060[1] 1788801659.649501 read 0x25c000 thru 0x260be9 (0x4bea bytes) 000062[0] 1788801659.677014 write_OD 0x4ea000 thru 0x4f1fff (0x8000 bytes) 000063[0] 1788801659.787632 trunc from 0x751a29 to 0x555dbf (0x1fbc69 bytes) 000064[1] 1788801659.886041 write 0x157000 thru 0x15846e (0x146f bytes) 000066[0] 1788801659.959346 write_OD 0x505000 thru 0x511fff (0xd000 bytes) 000068[0] 1788801660.061899 read_OD 0x00e000 thru 0x010fff (0x3000 bytes) 000070[1] 1788801660.073233 mapread 0x098000 thru 0x09fa30 (0x7a31 bytes) 000071[1] 1788801660.121444 trunc from 0x555dc0 to 0x4ce263 (0x87b5c bytes) 000073[1] 1788801660.264461 mapread 0x4bc000 thru 0x4c9c8c (0xdc8d bytes) 000074[1] 1788801660.336126 mapread 0x057000 thru 0x05f05b (0x805c bytes) 000075[1] 1788801660.382341 read 0x1f3000 thru 0x1f75e9 (0x45ea bytes) 000077[0] 1788801660.394064 write_OD 0x265000 thru 0x272fff (0xe000 bytes) 000078[1] 1788801660.546576 mapread 0x205000 thru 0x214893 (0xf894 bytes) 000079[1] 1788801660.598134 mapread 0x181000 thru 0x19094f (0xf950 bytes) 000082[1] 1788801660.644588 write 0x4d6000 thru 0x4d96f4 (0x36f5 bytes) 000083[1] 1788801660.691179 read 0x061000 thru 0x0664d1 (0x54d2 bytes) 000084[0] 1788801660.703576 punch from 0x109fe0 to 0x117ec3 (0xdee4 bytes) 000086[0] 1788801660.778155 punch from 0x1864b4 to 0x18c701 (0x624e bytes) 000087[1] 1788801660.808221 trunc from 0x4d96f5 to 0x78f452 (0x2b5d5e bytes) 000088[0] 1788801660.899410 mapread 0x6db000 thru 0x6e0bf6 (0x5bf7 bytes) 000090[1] 1788801660.985281 write 0x0ca000 thru 0x0d377c (0x977d bytes) 000091[1] 1788801661.024182 mapread 0x6c6000 thru 0x6cff7d (0x9f7e bytes) 000093[1] 1788801661.131261 mapread 0x64f000 thru 0x652feb (0x3fec bytes) 000095[1] 1788801661.237033 read 0x6ac000 thru 0x6b3b75 (0x7b76 bytes) 000096[1] 1788801661.262189 punch from 0x21a7f4 to 0x226074 (0xb881 bytes) 000098[0] 1788801661.296982 trunc from 0x78f453 to 0x6e8b1e (0xa6934 bytes) 000099[0] 1788801661.385518 write_OD 0x6e0000 thru 0x6e6fff (0x7000 bytes) 000100[0] 1788801661.514597 write_OD 0x5bb000 thru 0x5bdfff (0x3000 bytes) 000101[1] 1788801661.561453 write 0x97d000 thru 0x981e64 (0x4e65 bytes) 000102[1] 1788801661.573770 read 0x834000 thru 0x843e17 (0xfe18 bytes) 000104[0] 1788801661.617634 mapwrite 0x31f000 thru 0x32811c (0x911d bytes) 000105[0] 1788801661.724573 mapread 0x926000 thru 0x932dc2 (0xcdc3 bytes) 000106[0] 1788801661.758513 read_OD 0x67c000 thru 0x686fff (0xb000 bytes) 000107[1] 1788801661.786587 mapread 0x055000 thru 0x05d11e (0x811f bytes) 000110[1] 1788801661.830085 mapwrite 0x883000 thru 0x889fa8 (0x6fa9 bytes) 000113[0] 1788801661.949491 mapread 0x2bc000 thru 0x2c8c76 (0xcc77 bytes) 000114[0] 1788801662.040138 punch from 0x0d3e46 to 0x0dedcd (0xaf88 bytes) 000116[1] 1788801662.083152 mapwrite 0x2dc000 thru 0x2dd236 (0x1237 bytes) 000118[1] 1788801662.356500 write 0x3bd000 thru 0x3bff34 (0x2f35 bytes) 000119[1] 1788801662.373225 mapread 0x208000 thru 0x20e009 (0x600a bytes) 000121[1] 1788801662.392538 write 0x4c3000 thru 0x4cc5f9 (0x95fa bytes) 000124[0] 1788801662.462897 write_OD 0x864000 thru 0x86dfff (0xa000 bytes) 000125[0] 1788801662.608371 mapread 0x77e000 thru 0x78b753 (0xd754 bytes) 000126[0] 1788801662.668407 mapwrite 0x6e5000 thru 0x6e945e (0x445f bytes) 000127[0] 1788801662.927699 mapwrite 0x9cb000 thru 0x9d3996 (0x8997 bytes) 000128[1] 1788801663.168008 trunc from 0x9d3997 to 0x7dfcf1 (0x1f3ca5 bytes) 000129[0] 1788801663.221343 punch from 0x727f73 to 0x73260b (0xa699 bytes) 000131[0] 1788801663.254459 write_OD 0x49d000 thru 0x4a5fff (0x9000 bytes) 000132[0] 1788801663.350078 punch from 0x749051 to 0x751586 (0x8536 bytes) 000134[0] 1788801663.423709 read_OD 0x60d000 thru 0x613fff (0x7000 bytes) 000135[1] 1788801663.456022 mapread 0x2ae000 thru 0x2b7175 (0x9176 bytes) 000136[0] 1788801663.479818 write_OD 0x84a000 thru 0x857fff (0xe000 bytes) 000137[1] 1788801663.541340 read 0x7c1000 thru 0x7d029b (0xf29c bytes) 000139[0] 1788801663.553781 trunc from 0x858000 to 0x012acc (0x845533 bytes) 000141[0] 1788801663.630745 mapwrite 0x6a5000 thru 0x6a7437 (0x2438 bytes) 000143[0] 1788801663.771572 write_OD 0x981000 thru 0x986fff (0x6000 bytes) 000144[1] 1788801663.885648 punch from 0x428f7d to 0x432581 (0x9605 bytes) 000146[0] 1788801663.916448 mapwrite 0x597000 thru 0x5a0ccf (0x9cd0 bytes) 000147[1] 1788801664.057144 mapwrite 0x812000 thru 0x814827 (0x2828 bytes) 000149[1] 1788801664.170656 write 0x3ff000 thru 0x407e6e (0x8e6f bytes) 000150[0] 1788801664.181980 write_OD 0x7f1000 thru 0x7f7fff (0x7000 bytes) 000151[0] 1788801664.237349 mapwrite 0x2ed000 thru 0x2f129d (0x429e bytes) 000152[1] 1788801664.387440 punch from 0x0e8d58 to 0x0f63bd (0xd666 bytes) 000154[1] 1788801664.438887 mapwrite 0x9b5000 thru 0x9c0fb7 (0xbfb8 bytes) 000155[0] 1788801664.614957 mapwrite 0x41c000 thru 0x42bafe (0xfaff bytes) 000158[1] 1788801664.874135 write 0x1b7000 thru 0x1c5084 (0xe085 bytes) 000159[1] 1788801664.917334 mapwrite 0x352000 thru 0x35455a (0x255b bytes) 000160[1] 1788801664.996415 write 0x63e000 thru 0x641cef (0x3cf0 bytes) 000161[0] 1788801665.020201 trunc from 0x9c0fb8 to 0x0896cc (0x9378eb bytes) 000162[1] 1788801665.066076 read 0x005000 thru 0x00ac37 (0x5c38 bytes) 000163[0] 1788801665.086884 write_OD 0x363000 thru 0x36afff (0x8000 bytes) 000164[0] 1788801665.169269 trunc from 0x36b000 to 0x0ea219 (0x280de6 bytes) 000165[0] 1788801665.201706 write_OD 0x00b000 thru 0x016fff (0xc000 bytes) 000169[1] 1788801665.303397 read 0x064000 thru 0x065394 (0x1395 bytes) 000170[0] 1788801665.353546 mapwrite 0x55e000 thru 0x55e53c (0x53d bytes) 000171[1] 1788801665.526211 punch from 0x36e69a to 0x37b045 (0xc9ac bytes) 000172[0] 1788801665.564412 trunc from 0x55e53d to 0x1e4544 (0x379ff8 bytes) 000173[0] 1788801665.599608 mapread 0x03d000 thru 0x048d90 (0xbd91 bytes) 000174[1] 1788801665.625708 write 0x6fc000 thru 0x7033cd (0x73ce bytes) 000176[0] 1788801665.668240 mapread 0x6ad000 thru 0x6ad904 (0x905 bytes) 000177[0] 1788801665.703384 mapwrite 0x73c000 thru 0x741da9 (0x5daa bytes) 000178[1] 1788801665.882595 mapread 0x258000 thru 0x265032 (0xd033 bytes) 000179[1] 1788801666.019878 trunc from 0x741daa to 0x5e0ac9 (0x1612e0 bytes) 000181[1] 1788801666.055659 write 0x08b000 thru 0x08da6b (0x2a6c bytes) 000182[1] 1788801666.061719 mapread 0x3a2000 thru 0x3aa19d (0x819e bytes) 000183[1] 1788801666.133376 mapwrite 0x33f000 thru 0x34787d (0x887e bytes) 000184[1] 1788801666.218201 write 0x05a000 thru 0x061687 (0x7688 bytes) 000185[0] 1788801666.262560 read_OD 0x1f1000 thru 0x1f9fff (0x9000 bytes) 000186[0] 1788801666.329423 write_OD 0x9dc000 thru 0x9defff (0x3000 bytes) 000190[0] 1788801666.448443 punch from 0x606c51 to 0x608f63 (0x2313 bytes) 000192[0] 1788801666.474171 read_OD 0x1a3000 thru 0x1aefff (0xc000 bytes) 000193[1] 1788801666.490487 trunc from 0x9df000 to 0x356ebc (0x688143 bytes) 000196[0] 1788801666.546588 mapwrite 0x348000 thru 0x34de64 (0x5e65 bytes) 000197[1] 1788801666.619149 mapwrite 0x05f000 thru 0x061b40 (0x2b41 bytes) 000198[0] 1788801666.721781 write_OD 0x670000 thru 0x675fff (0x6000 bytes) truncating to largest ever: 0x9c9314 000199[1] 1788801666.805999 trunc from 0x676000 to 0x9c9313 (0x353314 bytes) 000200[0] 1788801666.849628 write_OD 0x95e000 thru 0x960fff (0x3000 bytes) 000201[0] 1788801666.902208 mapwrite 0x2da000 thru 0x2da05d (0x5e bytes) 000202[1] 1788801667.026976 trunc from 0x9c9314 to 0x3c3014 (0x6062ff bytes) 000203[0] 1788801667.089376 read_OD 0x19e000 thru 0x1abfff (0xe000 bytes) 000204[1] 1788801667.164869 mapread 0x1e2000 thru 0x1e9ba5 (0x7ba6 bytes) 000205[1] 1788801667.207299 trunc from 0x3c3015 to 0x0f6d52 (0x2cc2c2 bytes) 000206[1] 1788801667.250310 punch from 0x0f641b to 0x0f6d52 (0x938 bytes) 000207[1] 1788801667.271436 write 0x364000 thru 0x364a1d (0xa1e bytes) 000208[0] 1788801667.274156 mapread 0x22d000 thru 0x237c62 (0xac63 bytes) 000210[0] 1788801667.397137 read_OD 0x216000 thru 0x219fff (0x4000 bytes) 000211[0] 1788801667.405411 mapwrite 0x244000 thru 0x247985 (0x3986 bytes) 000212[1] 1788801667.572437 mapread 0x1d5000 thru 0x1da1f1 (0x51f2 bytes) 000213[0] 1788801667.664411 mapread 0x116000 thru 0x11c4a4 (0x64a5 bytes) 000216[0] 1788801667.700753 mapwrite 0x984000 thru 0x98c350 (0x8351 bytes) 000218[0] 1788801668.058636 punch from 0x672c9d to 0x67dee6 (0xb24a bytes) 000219[0] 1788801668.082853 punch from 0x618004 to 0x626ccd (0xecca bytes) 000220[1] 1788801668.087877 write 0x2ee000 thru 0x2ef337 (0x1338 bytes) 000222[1] 1788801668.130339 mapread 0x353000 thru 0x35abd6 (0x7bd7 bytes) 000223[0] 1788801668.182731 write_OD 0x81b000 thru 0x81cfff (0x2000 bytes) 000226[0] 1788801668.258447 mapwrite 0x40a000 thru 0x413122 (0x9123 bytes) 000228[1] 1788801668.422383 read 0x4e1000 thru 0x4e3e0c (0x2e0d bytes) 000229[0] 1788801668.451152 mapread 0x027000 thru 0x02a9b4 (0x39b5 bytes) 000230[1] 1788801668.476413 mapread 0x11a000 thru 0x11e9be (0x49bf bytes) 000231[1] 1788801668.499455 write 0x3e5000 thru 0x3e80ec (0x30ed bytes) 000233[1] 1788801668.541333 punch from 0x449c15 to 0x457ded (0xe1d9 bytes) 000239[0] 1788801668.633755 mapread 0x7c7000 thru 0x7d577b (0xe77c bytes) 000240[0] 1788801668.744422 punch from 0x0ef790 to 0x0fe1c2 (0xea33 bytes) 000242[1] 1788801668.761064 mapread 0x76e000 thru 0x77ab50 (0xcb51 bytes) 000243[1] 1788801668.818852 trunc from 0x98c351 to 0x652708 (0x339c48 bytes) 000245[1] 1788801668.927282 mapwrite 0x0b9000 thru 0x0c2dfe (0x9dff bytes) 000246[0] 1788801669.026696 read_OD 0x1c1000 thru 0x1cbfff (0xb000 bytes) 000247[0] 1788801669.050653 punch from 0x5930ab to 0x59a59f (0x74f5 bytes) 000248[1] 1788801669.081349 write 0x6ea000 thru 0x6ee950 (0x4951 bytes) 000249[1] 1788801669.108410 mapwrite 0x817000 thru 0x8172ec (0x2ed bytes) 000250[1] 1788801669.182156 mapread 0x76d000 thru 0x7742ff (0x7300 bytes) 000252[0] 1788801669.216811 punch from 0x269174 to 0x272b2d (0x99ba bytes) 000253[0] 1788801669.242318 trunc from 0x8172ed to 0x02bfed (0x7eb2ff bytes) 000254[0] 1788801669.299146 read_OD 0x01a000 thru 0x01ffff (0x6000 bytes) 000255[0] 1788801669.326818 mapread 0x00d000 thru 0x018972 (0xb973 bytes) 000256[0] 1788801669.347622 write_OD 0x130000 thru 0x13ffff (0x10000 bytes) 000257[1] 1788801669.479385 read 0x09d000 thru 0x0a1d87 (0x4d88 bytes) 000258[0] 1788801669.525840 trunc from 0x140000 to 0x431e64 (0x2f1e65 bytes) 000259[1] 1788801669.567021 trunc from 0x431e65 to 0x94e881 (0x51ca1d bytes) 000260[0] 1788801669.597130 punch from 0x61a952 to 0x61ea03 (0x40b2 bytes) 000261[0] 1788801669.633140 read_OD 0x8f3000 thru 0x8f5fff (0x3000 bytes) 000262[1] 1788801669.651555 read 0x744000 thru 0x747c9d (0x3c9e bytes) 000264[1] 1788801669.666056 mapwrite 0x755000 thru 0x755717 (0x718 bytes) 000265[1] 1788801669.806139 read 0x5c4000 thru 0x5d3b61 (0xfb62 bytes) 000267[1] 1788801669.842488 read 0x20c000 thru 0x21279a (0x679b bytes) 000269[1] 1788801669.852839 trunc from 0x94e882 to 0x7badb9 (0x193ac8 bytes) 000270[1] 1788801669.879801 mapread 0x421000 thru 0x42c76b (0xb76c bytes) 000274[0] 1788801669.922415 punch from 0x2e3ff8 to 0x2f0f65 (0xcf6e bytes) 000275[1] 1788801669.951819 read 0x3d3000 thru 0x3d3138 (0x139 bytes) 000276[0] 1788801669.975029 mapread 0x24c000 thru 0x25463e (0x863f bytes) 000279[1] 1788801669.998301 punch from 0x61e551 to 0x6243b3 (0x5e63 bytes) 000280[0] 1788801670.014446 punch from 0x4ef804 to 0x4f660c (0x6e09 bytes) 000284[1] 1788801670.081842 mapread 0x612000 thru 0x61453b (0x253c bytes) 000286[1] 1788801670.145260 mapread 0x4cd000 thru 0x4dc102 (0xf103 bytes) 000290[0] 1788801670.178050 punch from 0x782554 to 0x78746f (0x4f1c bytes) 000291[1] 1788801670.194334 punch from 0x4d7569 to 0x4dbed2 (0x496a bytes) 000292[1] 1788801670.221122 punch from 0x52f636 to 0x5310ca (0x1a95 bytes) 000293[0] 1788801670.228454 write_OD 0x64e000 thru 0x65afff (0xd000 bytes) 000296[0] 1788801670.280772 mapread 0x0dd000 thru 0x0e586e (0x886f bytes) 000297[1] 1788801670.298831 mapread 0x263000 thru 0x266478 (0x3479 bytes) 000298[0] 1788801670.322414 punch from 0x522e4d to 0x53056b (0xd71f bytes) 000299[1] 1788801670.350392 mapread 0x722000 thru 0x72e0a1 (0xc0a2 bytes) 000301[1] 1788801670.416356 mapread 0x126000 thru 0x1351cf (0xf1d0 bytes) 000302[1] 1788801670.442683 trunc from 0x7badba to 0x7cfc28 (0x14e6f bytes) 000305[0] 1788801670.511295 write_OD 0x664000 thru 0x66bfff (0x8000 bytes) 000308[0] 1788801670.575625 mapread 0x1f1000 thru 0x2000e5 (0xf0e6 bytes) 000309[0] 1788801670.650801 punch from 0x74e21b to 0x7528b0 (0x4696 bytes) 000310[0] 1788801670.680640 trunc from 0x7cfc29 to 0x83fb2e (0x6ff06 bytes) 000313[0] 1788801670.785391 read_OD 0x060000 thru 0x068fff (0x9000 bytes) 000316[1] 1788801670.815375 write 0x637000 thru 0x63a0d7 (0x30d8 bytes) 000317[1] 1788801670.837545 trunc from 0x83fb2f to 0x56ca92 (0x2d309c bytes) 000318[1] 1788801670.884583 punch from 0x1cdaa6 to 0x1d049b (0x29f6 bytes) 000320[1] 1788801670.895663 trunc from 0x56ca93 to 0x66eff1 (0x10255f bytes) 000321[0] 1788801670.965870 punch from 0x393de4 to 0x3a1fc3 (0xe1e0 bytes) 000322[1] 1788801671.001429 trunc from 0x66eff2 to 0x6d6a82 (0x67a91 bytes) 000324[0] 1788801671.089643 trunc from 0x6d6a83 to 0x21fc8f (0x4b6df3 bytes) 000325[0] 1788801671.143764 trunc from 0x21fc90 to 0x1e6f2d (0x38d62 bytes) 000326[1] 1788801671.191372 mapwrite 0x522000 thru 0x52542b (0x342c bytes) 000328[1] 1788801671.342751 write 0x48b000 thru 0x497de5 (0xcde6 bytes) 000331[0] 1788801671.401546 write_OD 0x1be000 thru 0x1c7fff (0xa000 bytes) 000332[1] 1788801671.498181 read 0x28f000 thru 0x29c899 (0xd89a bytes) 000335[0] 1788801671.519383 mapwrite 0x53b000 thru 0x53cdb7 (0x1db8 bytes) 000336[0] 1788801671.663498 write_OD 0x0ce000 thru 0x0d1fff (0x4000 bytes) 000339[1] 1788801671.814833 punch from 0x367389 to 0x36992e (0x25a6 bytes) 000341[0] 1788801671.888690 mapwrite 0x7d7000 thru 0x7e5bea (0xebeb bytes) 000342[0] 1788801672.096680 read_OD 0x12a000 thru 0x138fff (0xf000 bytes) 000343[0] 1788801672.114180 trunc from 0x7e5beb to 0x5d1efa (0x213cf0 bytes) 000344[0] 1788801672.178091 trunc from 0x5d1efb to 0x444140 (0x18ddba bytes) 000347[0] 1788801672.254547 punch from 0x2f02c1 to 0x2f62b9 (0x5ff9 bytes) 000348[1] 1788801672.294520 mapread 0x134000 thru 0x1348e2 (0x8e3 bytes) 000353[0] 1788801672.320909 trunc from 0x444141 to 0x1066b1 (0x33da8f bytes) 000355[0] 1788801672.473668 punch from 0x02e1fd to 0x03b736 (0xd53a bytes) 000358[1] 1788801672.496508 punch from 0x06feb6 to 0x074806 (0x4951 bytes) 000360[0] 1788801672.566359 read_OD 0x0ac000 thru 0x0bafff (0xf000 bytes) 000363[1] 1788801672.614436 trunc from 0x1066b2 to 0x3126f1 (0x20c040 bytes) 000365[0] 1788801672.745792 mapread 0x2de000 thru 0x2e61ae (0x81af bytes) 000366[0] 1788801672.928999 read_OD 0x09d000 thru 0x09efff (0x2000 bytes) truncating to largest ever: 0x9fb1c5 000367[0] 1788801672.954130 trunc from 0x3126f2 to 0x9fb1c4 (0x6e8ad3 bytes) 000368[1] 1788801673.037730 trunc from 0x9fb1c5 to 0x9b01e8 (0x4afdc bytes) 000369[1] 1788801673.081110 write 0x893000 thru 0x89ad45 (0x7d46 bytes) 000370[1] 1788801673.110390 mapwrite 0x17b000 thru 0x184865 (0x9866 bytes) 000371[0] 1788801673.213076 punch from 0x5bc153 to 0x5c2cf1 (0x6b9f bytes) 000374[0] 1788801673.292385 punch from 0x2a21b7 to 0x2af0cb (0xcf15 bytes) 000376[1] 1788801673.300231 mapread 0x31c000 thru 0x31c31a (0x31b bytes) 000378[0] 1788801673.339334 trunc from 0x9b01e9 to 0x79fbb6 (0x210632 bytes) 000382[0] 1788801673.407223 read_OD 0x3fd000 thru 0x401fff (0x5000 bytes) 000384[1] 1788801673.438313 mapwrite 0x95e000 thru 0x969e94 (0xbe95 bytes) 000386[0] 1788801673.557129 mapwrite 0x36c000 thru 0x36ecfd (0x2cfe bytes) 000389[0] 1788801673.640764 mapread 0x465000 thru 0x46825a (0x325b bytes) 000392[0] 1788801673.666678 write_OD 0x5aa000 thru 0x5acfff (0x3000 bytes) 000393[1] 1788801673.716151 mapwrite 0x774000 thru 0x77a041 (0x6042 bytes) 000394[1] 1788801673.815789 mapread 0x314000 thru 0x31cfb4 (0x8fb5 bytes) 000396[0] 1788801673.877565 punch from 0x5b3bd5 to 0x5bc38a (0x87b6 bytes) 000398[1] 1788801673.921488 mapwrite 0x311000 thru 0x31c0f0 (0xb0f1 bytes) 000399[0] 1788801674.050846 punch from 0x83dd59 to 0x84af52 (0xd1fa bytes) 000400[1] 1788801674.082507 mapwrite 0x832000 thru 0x83b9ca (0x99cb bytes) 000403[1] 1788801674.211324 trunc from 0x969e95 to 0x807d87 (0x16210d bytes) 000405[1] 1788801674.281414 punch from 0x1fa27e to 0x1fff20 (0x5ca3 bytes) 000406[0] 1788801674.309621 trunc from 0x807d88 to 0x6eb564 (0x11c823 bytes) 000407[0] 1788801674.360575 read_OD 0x0fb000 thru 0x0fffff (0x5000 bytes) 000409[1] 1788801674.401555 punch from 0x474aa9 to 0x481cbc (0xd214 bytes) 000410[0] 1788801674.417334 mapwrite 0x3db000 thru 0x3e3049 (0x804a bytes) 000413[0] 1788801674.628973 write_OD 0x160000 thru 0x161fff (0x2000 bytes) 000415[1] 1788801674.749126 mapread 0x247000 thru 0x255120 (0xe121 bytes) 000417[1] 1788801674.821665 mapwrite 0x91e000 thru 0x921d43 (0x3d44 bytes) 000420[1] 1788801674.994299 read 0x33d000 thru 0x349195 (0xc196 bytes) 000422[0] 1788801675.045818 trunc from 0x921d44 to 0x86e541 (0xb3802 bytes) 000423[1] 1788801675.115479 punch from 0x5b0369 to 0x5bd0ba (0xcd52 bytes) 000424[1] 1788801675.146899 trunc from 0x86e542 to 0x5b903d (0x2b5504 bytes) 000425[1] 1788801675.168567 mapwrite 0x4d7000 thru 0x4defb3 (0x7fb4 bytes) 000426[0] 1788801675.258320 trunc from 0x5b903e to 0x8ed9b9 (0x33497c bytes) 000429[1] 1788801675.338336 punch from 0x5273ac to 0x52831f (0xf74 bytes) 000432[1] 1788801675.426689 trunc from 0x8ed9ba to 0x58b115 (0x3628a4 bytes) 000433[0] 1788801675.446489 write_OD 0x34b000 thru 0x351fff (0x7000 bytes) 000434[1] 1788801675.502869 mapwrite 0x298000 thru 0x29bbea (0x3beb bytes) 000435[1] 1788801675.589695 write 0x157000 thru 0x15dd5b (0x6d5c bytes) 000436[1] 1788801675.613352 write 0x794000 thru 0x7a3b70 (0xfb71 bytes) 000443[1] 1788801675.666998 punch from 0x4db41d to 0x4e2d83 (0x7967 bytes) 000444[1] 1788801675.673203 write 0x7d6000 thru 0x7e5779 (0xf77a bytes) 000445[0] 1788801675.675500 punch from 0x7c6a9d to 0x7c7a4d (0xfb1 bytes) 000447[1] 1788801675.742209 punch from 0x1932e3 to 0x19a1a9 (0x6ec7 bytes) 000448[0] 1788801675.761142 read_OD 0x362000 thru 0x36cfff (0xb000 bytes) 000449[0] 1788801675.791826 mapread 0x3a3000 thru 0x3a575f (0x2760 bytes) 000451[1] 1788801675.842538 trunc from 0x7e577a to 0x961b66 (0x17c3ed bytes) 000453[1] 1788801675.925063 mapwrite 0x264000 thru 0x26ea98 (0xaa99 bytes) 000454[1] 1788801676.032923 mapread 0x915000 thru 0x91f590 (0xa591 bytes) 000455[1] 1788801676.069056 read 0x7cf000 thru 0x7de5c0 (0xf5c1 bytes) 000456[0] 1788801676.098070 read_OD 0x001000 thru 0x001fff (0x1000 bytes) 000457[0] 1788801676.124213 mapwrite 0x1fe000 thru 0x20c07c (0xe07d bytes) 000461[1] 1788801676.324827 punch from 0x88792c to 0x893ec5 (0xc59a bytes) 000462[1] 1788801676.352625 trunc from 0x961b67 to 0x84fa05 (0x112161 bytes) 000463[1] 1788801676.384390 read 0x09f000 thru 0x0a54ae (0x64af bytes) 000464[0] 1788801676.404376 mapread 0x4d9000 thru 0x4e45e8 (0xb5e9 bytes) 000465[0] 1788801676.476437 write_OD 0x5f0000 thru 0x5fbfff (0xc000 bytes) 000467[1] 1788801676.582351 punch from 0x4293f8 to 0x436f20 (0xdb29 bytes) 000468[0] 1788801676.600895 trunc from 0x84fa06 to 0x6c25ac (0x18d459 bytes) 000470[0] 1788801676.629837 read_OD 0x43b000 thru 0x448fff (0xe000 bytes) 000472[0] 1788801676.653414 mapwrite 0x129000 thru 0x133820 (0xa821 bytes) 000473[0] 1788801676.809843 read_OD 0x3d2000 thru 0x3dbfff (0xa000 bytes) 000474[1] 1788801676.852624 punch from 0x054fd7 to 0x058d21 (0x3d4b bytes) 000476[1] 1788801676.954525 punch from 0x51aa8c to 0x51d75e (0x2cd3 bytes) 000477[0] 1788801676.974654 mapread 0x1d5000 thru 0x1e0a3a (0xba3b bytes) 000478[0] 1788801677.078357 mapwrite 0x453000 thru 0x460ab4 (0xdab5 bytes) 000479[1] 1788801677.229898 punch from 0x1f7998 to 0x202684 (0xaced bytes) 000483[1] 1788801677.290333 trunc from 0x6c25ad to 0x907a98 (0x2454ec bytes) 000484[1] 1788801677.344343 mapread 0x746000 thru 0x754706 (0xe707 bytes) 000486[0] 1788801677.372860 trunc from 0x907a99 to 0x30bac7 (0x5fbfd1 bytes) 000487[1] 1788801677.419226 mapwrite 0x713000 thru 0x71e069 (0xb06a bytes) 000488[0] 1788801677.530775 read_OD 0x017000 thru 0x024fff (0xe000 bytes) 000489[1] 1788801677.570614 mapwrite 0x484000 thru 0x48e1f2 (0xa1f3 bytes) 000490[0] 1788801677.708228 trunc from 0x71e06a to 0x2c5905 (0x458764 bytes) 000492[0] 1788801677.776858 read_OD 0x03e000 thru 0x03efff (0x1000 bytes) 000493[1] 1788801677.792983 trunc from 0x2c5906 to 0x72bf92 (0x46668d bytes) 000494[0] 1788801677.845869 mapread 0x45a000 thru 0x46580e (0xb80f bytes) 000495[1] 1788801678.007688 mapwrite 0x6ec000 thru 0x6f0d1c (0x4d1d bytes) 000496[1] 1788801678.130886 read 0x5eb000 thru 0x5f8bf7 (0xdbf8 bytes) 000499[0] 1788801678.268565 read_OD 0x13f000 thru 0x14dfff (0xf000 bytes) 000500[1] 1788801678.306742 mapwrite 0x44a000 thru 0x44b2cd (0x12ce bytes) All operations completed A-OK! PASS 16a (63s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 13:21:23 (1788801683) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.314878 s, 26.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.3222 s, 26.0 MB/s Chance of close/open is 1 in 50 Seed set to 1687 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1788801688.482528 mapwrite 0x475176 thru 0x476385 (0x1210 bytes) truncating to largest ever: 0x7d0255 000003[0] 1788801688.617782 trunc from 0x476386 to 0x7d0254 (0x359ecf bytes) 000006[0] 1788801688.712717 read 0x25f90c thru 0x261813 (0x1f08 bytes) 000007[1] 1788801688.719452 mapwrite 0x0f681b thru 0x0f8368 (0x1b4e bytes) 000008[0] 1788801688.852849 trunc from 0x7d0255 to 0x6fad47 (0xd550d bytes) 000009[1] 1788801688.907856 punch from 0x5d0bc3 to 0x5d170c (0xb4a bytes) 000010[1] 1788801688.946708 mapwrite 0x4e8c39 thru 0x4e9b21 (0xee9 bytes) 000012[1] 1788801689.033127 read 0x63db8d thru 0x63e570 (0x9e4 bytes) 000013[1] 1788801689.059203 write 0x5cfaa2 thru 0x5cfb8c (0xeb bytes) 000015[1] 1788801689.103393 trunc from 0x6fad48 to 0x755945 (0x5abfe bytes) 000017[1] 1788801689.136456 write 0x8327fa thru 0x8332a8 (0xaaf bytes) 000018[1] 1788801689.164714 read 0x1b3180 thru 0x1b3b33 (0x9b4 bytes) 000019[0] 1788801689.171446 trunc from 0x8332a9 to 0x1b44e0 (0x67edc8 bytes) 000020[1] 1788801689.250473 trunc from 0x1b44e1 to 0x254e0c (0xa092c bytes) 000021[0] 1788801689.302247 mapread 0x08f51c thru 0x0907f4 (0x12d9 bytes) 000023[0] 1788801689.339330 read 0x243b6c thru 0x2456a2 (0x1b37 bytes) 000024[1] 1788801689.388189 mapread 0x1396df thru 0x13b09f (0x19c1 bytes) 000025[0] 1788801689.417404 punch from 0x039b0c to 0x03b3cc (0x18c1 bytes) 000026[1] 1788801689.448379 read 0x11f12f thru 0x120702 (0x15d4 bytes) 000027[1] 1788801689.531664 punch from 0x04a7a7 to 0x04b976 (0x11d0 bytes) 000028[0] 1788801689.570988 read 0x05b93c thru 0x05c811 (0xed6 bytes) 000030[0] 1788801689.656472 mapwrite 0x1773f6 thru 0x178f96 (0x1ba1 bytes) 000031[1] 1788801689.775402 write 0x457fb3 thru 0x458982 (0x9d0 bytes) 000033[1] 1788801689.812573 mapread 0x1b859e thru 0x1b8c76 (0x6d9 bytes) 000034[1] 1788801689.859543 mapread 0x26a39d thru 0x26b71b (0x137f bytes) 000035[1] 1788801689.887357 mapwrite 0x933ab4 thru 0x9354d1 (0x1a1e bytes) 000038[0] 1788801690.289603 write 0x6552db thru 0x65710f (0x1e35 bytes) 000039[1] 1788801690.365473 punch from 0x1039c3 to 0x105566 (0x1ba4 bytes) 000040[0] 1788801690.390323 mapwrite 0x365dc5 thru 0x3670d8 (0x1314 bytes) 000046[1] 1788801690.643429 read 0x87557c thru 0x876fd5 (0x1a5a bytes) 000047[1] 1788801690.667446 mapread 0x095840 thru 0x096d4d (0x150e bytes) 000051[1] 1788801690.731443 mapwrite 0x15ef7a thru 0x160539 (0x15c0 bytes) 000053[1] 1788801690.902462 mapwrite 0x37342a thru 0x374273 (0xe4a bytes) 000056[0] 1788801691.108490 write 0x8f786b thru 0x8f7c6f (0x405 bytes) 000060[0] 1788801691.248839 mapread 0x12cc0e thru 0x12e90a (0x1cfd bytes) 000061[1] 1788801691.310481 mapread 0x1e0244 thru 0x1e18ba (0x1677 bytes) 000065[1] 1788801691.494837 mapread 0x1af3ad thru 0x1b08c5 (0x1519 bytes) 000069[1] 1788801691.524498 read 0x09dcf7 thru 0x09ed48 (0x1052 bytes) truncating to largest ever: 0x975196 000070[1] 1788801691.530380 trunc from 0x9354d2 to 0x975195 (0x3fcc4 bytes) 000072[0] 1788801691.615316 write 0x1d6257 thru 0x1d7025 (0xdcf bytes) 000073[1] 1788801691.655332 trunc from 0x975196 to 0x5f1eba (0x3832db bytes) 000074[0] 1788801691.690457 trunc from 0x5f1ebb to 0x649e11 (0x57f57 bytes) 000078[1] 1788801691.792357 write 0x17c48f thru 0x17cf98 (0xb0a bytes) 000082[1] 1788801691.869401 mapread 0x16495a thru 0x1665e2 (0x1c89 bytes) 000084[0] 1788801691.931742 trunc from 0x649e12 to 0x431ed8 (0x217f39 bytes) 000085[1] 1788801691.978370 read 0x1bf9ec thru 0x1c0358 (0x96d bytes) 000086[0] 1788801692.020619 trunc from 0x431ed9 to 0x7a215a (0x370282 bytes) 000091[1] 1788801692.196628 trunc from 0x7a215b to 0x851205 (0xaf0ab bytes) 000092[0] 1788801692.301433 mapread 0x540112 thru 0x54146d (0x135c bytes) 000094[0] 1788801692.391852 mapwrite 0x9d090c thru 0x9d1968 (0x105d bytes) 000095[0] 1788801692.492199 write 0x2d3d94 thru 0x2d5705 (0x1972 bytes) 000096[1] 1788801692.532296 mapwrite 0x028ff1 thru 0x02ad06 (0x1d16 bytes) 000097[1] 1788801692.729717 mapwrite 0x6bfafa thru 0x6c08b4 (0xdbb bytes) 000098[1] 1788801692.882651 write 0x548fd3 thru 0x54a9e9 (0x1a17 bytes) 000099[0] 1788801692.936737 trunc from 0x9d1969 to 0x2c2498 (0x70f4d0 bytes) 000100[1] 1788801693.059190 trunc from 0x2c2499 to 0x449b07 (0x18766f bytes) 000101[1] 1788801693.168663 punch from 0x216045 to 0x216227 (0x1e3 bytes) 000102[1] 1788801693.187903 mapread 0x12219c thru 0x123742 (0x15a7 bytes) 000103[1] 1788801693.214492 write 0x3d21ab thru 0x3d2429 (0x27f bytes) 000104[1] 1788801693.225596 punch from 0x2904d8 to 0x292399 (0x1ec2 bytes) 000107[0] 1788801693.282459 mapread 0x203bd2 thru 0x2056a0 (0x1acf bytes) 000108[0] 1788801693.376195 trunc from 0x449b08 to 0x2801e6 (0x1c9921 bytes) 000109[1] 1788801693.497077 trunc from 0x2801e7 to 0x67998c (0x3f97a6 bytes) 000112[1] 1788801693.710898 trunc from 0x67998d to 0x14157a (0x538412 bytes) 000113[1] 1788801693.748886 write 0x4c8fad thru 0x4c9f6e (0xfc2 bytes) 000114[0] 1788801693.759652 write 0x19a797 thru 0x19c6bd (0x1f27 bytes) 000115[0] 1788801693.827758 read 0x27523d thru 0x276e2f (0x1bf3 bytes) 000117[0] 1788801693.891460 punch from 0x18c90e to 0x18cc34 (0x327 bytes) 000118[0] 1788801693.914242 mapwrite 0x3ae285 thru 0x3af0a5 (0xe21 bytes) 000119[1] 1788801694.033866 read 0x13f8db thru 0x13ff58 (0x67e bytes) 000121[0] 1788801694.116429 mapread 0x459e03 thru 0x45b0fd (0x12fb bytes) 000122[1] 1788801694.190290 mapwrite 0x621f84 thru 0x623db4 (0x1e31 bytes) 000123[1] 1788801694.418084 write 0x884a08 thru 0x886073 (0x166c bytes) 000124[1] 1788801694.428176 write 0x25cd81 thru 0x25cdf2 (0x72 bytes) 000127[0] 1788801694.457148 write 0x27897c thru 0x279bca (0x124f bytes) 000129[0] 1788801694.517591 read 0x46b32c thru 0x46c2f4 (0xfc9 bytes) 000132[0] 1788801694.553602 read 0x7e643e thru 0x7e6e10 (0x9d3 bytes) 000135[1] 1788801694.613171 mapwrite 0x3e038a thru 0x3e1e11 (0x1a88 bytes) 000136[1] 1788801694.683291 trunc from 0x886074 to 0x4b3366 (0x3d2d0d bytes) 000138[0] 1788801694.761701 mapwrite 0x33e452 thru 0x340163 (0x1d12 bytes) 000141[0] 1788801694.964851 write 0x609480 thru 0x60a93e (0x14bf bytes) 000144[0] 1788801695.043245 mapwrite 0x5a753d thru 0x5a814d (0xc11 bytes) 000145[1] 1788801695.122041 write 0x46e5a1 thru 0x46f793 (0x11f3 bytes) 000148[1] 1788801695.188532 read 0x20faa4 thru 0x210d34 (0x1291 bytes) 000149[0] 1788801695.248373 mapread 0x068e7e thru 0x068e99 (0x1c bytes) 000150[1] 1788801695.281309 write 0x13ec75 thru 0x140967 (0x1cf3 bytes) 000152[0] 1788801695.410549 read 0x0bf7ea thru 0x0c094c (0x1163 bytes) 000154[1] 1788801695.453904 write 0x08ff48 thru 0x0906a1 (0x75a bytes) 000156[1] 1788801695.498666 mapwrite 0x090185 thru 0x091be4 (0x1a60 bytes) 000157[0] 1788801695.621247 mapwrite 0x7f9f26 thru 0x7fbcb6 (0x1d91 bytes) 000161[1] 1788801695.949624 punch from 0x6f9757 to 0x6fa1ed (0xa97 bytes) 000162[1] 1788801696.006104 mapread 0x4aeeb9 thru 0x4b0cad (0x1df5 bytes) 000163[1] 1788801696.069804 write 0x95679a thru 0x957e6b (0x16d2 bytes) 000164[0] 1788801696.113167 write 0x7c6c89 thru 0x7c6e92 (0x20a bytes) 000165[1] 1788801696.154281 read 0x725808 thru 0x726a07 (0x1200 bytes) 000167[1] 1788801696.194653 write 0x2673e7 thru 0x267774 (0x38e bytes) 000168[0] 1788801696.204835 read 0x830894 thru 0x830dbf (0x52c bytes) 000170[1] 1788801696.255693 mapwrite 0x173bc7 thru 0x17487c (0xcb6 bytes) 000171[0] 1788801696.398777 mapread 0x0354d6 thru 0x036a90 (0x15bb bytes) 000172[1] 1788801696.445172 punch from 0x0829c9 to 0x082b37 (0x16f bytes) 000174[1] 1788801696.488851 punch from 0x0526f8 to 0x054652 (0x1f5b bytes) 000175[0] 1788801696.505282 read 0x953a66 thru 0x95521d (0x17b8 bytes) 000176[0] 1788801696.521621 read 0x0831c5 thru 0x0844ca (0x1306 bytes) 000178[0] 1788801696.532343 trunc from 0x957e6c to 0x4c092e (0x49753d bytes) 000179[0] 1788801696.604851 punch from 0x18d5a6 to 0x18d7bc (0x217 bytes) 000180[0] 1788801696.614635 read 0x38ec19 thru 0x38f0e9 (0x4d1 bytes) 000181[1] 1788801696.663754 mapread 0x19ce23 thru 0x19eac3 (0x1ca1 bytes) 000182[0] 1788801696.742549 mapread 0x4a24f0 thru 0x4a3b44 (0x1655 bytes) 000183[0] 1788801696.805733 read 0x16e819 thru 0x16fb62 (0x134a bytes) 000184[0] 1788801696.821989 trunc from 0x4c092f to 0x6caa17 (0x20a0e9 bytes) 000186[0] 1788801696.903634 punch from 0x1c094c to 0x1c0a63 (0x118 bytes) 000187[0] 1788801696.907862 write 0x2dda6c thru 0x2ddaa2 (0x37 bytes) 000188[0] 1788801696.971898 punch from 0x56fb80 to 0x5710c2 (0x1543 bytes) 000189[1] 1788801696.977506 punch from 0x164666 to 0x1652f3 (0xc8e bytes) 000191[0] 1788801697.047725 mapread 0x3adf19 thru 0x3ae424 (0x50c bytes) 000193[0] 1788801697.135790 mapread 0x69f779 thru 0x69fd15 (0x59d bytes) 000195[1] 1788801697.213398 write 0x2692dd thru 0x269783 (0x4a7 bytes) 000197[1] 1788801697.286413 punch from 0x3cb9d2 to 0x3cc9f5 (0x1024 bytes) 000198[1] 1788801697.302409 mapread 0x0150cf thru 0x016c46 (0x1b78 bytes) 000205[0] 1788801697.499647 mapread 0x2389b0 thru 0x23a0b6 (0x1707 bytes) 000206[1] 1788801697.562711 read 0x0f6c07 thru 0x0f861a (0x1a14 bytes) 000209[0] 1788801697.595429 mapread 0x2b9ebc thru 0x2bb627 (0x176c bytes) 000210[0] 1788801697.628555 mapread 0x5c55c0 thru 0x5c74bc (0x1efd bytes) 000211[0] 1788801697.672629 punch from 0x090dc3 to 0x092235 (0x1473 bytes) 000212[0] 1788801697.698652 punch from 0x3fedf6 to 0x3ff311 (0x51c bytes) 000214[0] 1788801697.730569 mapread 0x3157c7 thru 0x31602a (0x864 bytes) 000215[1] 1788801697.773519 write 0x6aee2d thru 0x6af99f (0xb73 bytes) 000216[1] 1788801697.853341 punch from 0x41cf33 to 0x41cff0 (0xbe bytes) 000219[0] 1788801697.957355 trunc from 0x6caa18 to 0x81b884 (0x150e6d bytes) 000220[1] 1788801698.009661 write 0x9bf030 thru 0x9c046a (0x143b bytes) 000221[1] 1788801698.043609 mapread 0x7e3718 thru 0x7e4b6c (0x1455 bytes) 000223[1] 1788801698.079241 mapread 0x0f9596 thru 0x0f99eb (0x456 bytes) 000224[1] 1788801698.109908 punch from 0x1129cc to 0x1147d0 (0x1e05 bytes) 000226[1] 1788801698.117528 read 0x5d0c1f thru 0x5d10d1 (0x4b3 bytes) 000227[1] 1788801698.133184 mapread 0x9b0128 thru 0x9b0b97 (0xa70 bytes) 000230[0] 1788801698.161067 mapwrite 0x9b5350 thru 0x9b6f42 (0x1bf3 bytes) 000232[0] 1788801698.277193 punch from 0x8a0747 to 0x8a1370 (0xc2a bytes) 000235[0] 1788801698.336425 mapwrite 0x4b49e9 thru 0x4b56ca (0xce2 bytes) 000236[0] 1788801698.487352 write 0x5b82d6 thru 0x5b96e7 (0x1412 bytes) 000237[0] 1788801698.497657 read 0x3ccf9c thru 0x3cd8de (0x943 bytes) 000238[1] 1788801698.503175 mapwrite 0x103a78 thru 0x103fce (0x557 bytes) 000239[1] 1788801698.592405 punch from 0x883f26 to 0x8849be (0xa99 bytes) 000240[1] 1788801698.615469 mapread 0x6cd192 thru 0x6cdd24 (0xb93 bytes) 000242[1] 1788801698.736749 mapread 0x218ce0 thru 0x21a653 (0x1974 bytes) 000243[1] 1788801698.754509 write 0x5866d1 thru 0x5874fa (0xe2a bytes) 000247[1] 1788801698.797127 read 0x5375a8 thru 0x537a33 (0x48c bytes) 000250[0] 1788801698.824459 mapwrite 0x947da8 thru 0x948fc0 (0x1219 bytes) 000251[0] 1788801698.924734 trunc from 0x9c046b to 0x1d3528 (0x7ecf42 bytes) 000252[0] 1788801698.984476 read 0x08c0c5 thru 0x08cca2 (0xbde bytes) 000254[1] 1788801699.051711 mapread 0x1a47d5 thru 0x1a49f9 (0x225 bytes) 000255[1] 1788801699.095613 read 0x043501 thru 0x0441f4 (0xcf4 bytes) 000256[1] 1788801699.126341 read 0x143ea9 thru 0x14443f (0x597 bytes) 000262[1] 1788801699.213362 mapwrite 0x598b69 thru 0x59aa63 (0x1efb bytes) 000264[0] 1788801699.515805 punch from 0x3ceda1 to 0x3cefb3 (0x213 bytes) 000267[1] 1788801699.584529 write 0x34fa8d thru 0x3518b2 (0x1e26 bytes) 000268[0] 1788801699.622690 mapwrite 0x5ffaed thru 0x601015 (0x1529 bytes) 000269[1] 1788801699.959827 mapread 0x4c913e thru 0x4c917c (0x3f bytes) 000271[1] 1788801700.021528 read 0x410948 thru 0x412766 (0x1e1f bytes) 000272[1] 1788801700.039556 read 0x2b3697 thru 0x2b485f (0x11c9 bytes) 000273[1] 1788801700.072613 mapread 0x147f64 thru 0x149a98 (0x1b35 bytes) 000274[1] 1788801700.104709 mapread 0x1c835d thru 0x1c9692 (0x1336 bytes) 000275[0] 1788801700.116215 read 0x123a04 thru 0x123e39 (0x436 bytes) 000276[0] 1788801700.121305 trunc from 0x601016 to 0x581e6d (0x7f1a8 bytes) 000278[1] 1788801700.177223 read 0x41bc6f thru 0x41d536 (0x18c8 bytes) 000279[1] 1788801700.213153 read 0x181017 thru 0x1810d2 (0xbc bytes) 000280[1] 1788801700.234040 write 0x2de332 thru 0x2df6c4 (0x1393 bytes) 000281[0] 1788801700.289812 mapread 0x15da2f thru 0x15f48d (0x1a5f bytes) 000282[0] 1788801700.373880 trunc from 0x581e6e to 0x210dae (0x3710bf bytes) 000283[1] 1788801700.449172 mapread 0x1068bb thru 0x107d22 (0x1468 bytes) 000284[1] 1788801700.537568 read 0x14bf3d thru 0x14dc83 (0x1d47 bytes) 000285[1] 1788801700.620286 punch from 0x09fdae to 0x0a0115 (0x368 bytes) 000287[0] 1788801700.667797 mapwrite 0x684345 thru 0x685637 (0x12f3 bytes) 000288[0] 1788801700.757962 trunc from 0x685638 to 0x81d798 (0x198161 bytes) 000291[0] 1788801700.908036 trunc from 0x81d799 to 0x55ec59 (0x2beb3f bytes) 000292[1] 1788801700.964697 mapread 0x26b5bf thru 0x26c504 (0xf46 bytes) 000294[1] 1788801701.075430 read 0x506007 thru 0x5077a3 (0x179d bytes) 000295[1] 1788801701.100532 trunc from 0x55ec5a to 0x69663e (0x1379e5 bytes) 000297[1] 1788801701.255797 mapwrite 0x2d9fe8 thru 0x2da2af (0x2c8 bytes) 000300[1] 1788801701.397494 trunc from 0x69663f to 0x17ded6 (0x518768 bytes) 000302[1] 1788801701.428200 write 0x2dca80 thru 0x2de47b (0x19fc bytes) 000303[0] 1788801701.431578 read 0x20346b thru 0x204261 (0xdf7 bytes) 000304[0] 1788801701.484350 write 0x489395 thru 0x48b29b (0x1f07 bytes) 000305[1] 1788801701.515603 punch from 0x37ef28 to 0x38075f (0x1838 bytes) 000306[1] 1788801701.559632 read 0x36fc57 thru 0x3713b8 (0x1762 bytes) 000307[1] 1788801701.621523 punch from 0x2769c5 to 0x278359 (0x1995 bytes) 000308[1] 1788801701.656791 read 0x353646 thru 0x35453d (0xef8 bytes) 000310[0] 1788801701.736435 mapwrite 0x31e35f thru 0x3202ae (0x1f50 bytes) 000311[0] 1788801701.853860 mapwrite 0x763dca thru 0x7651ef (0x1426 bytes) 000314[1] 1788801701.971625 read 0x45d238 thru 0x45df4b (0xd14 bytes) 000316[1] 1788801702.001106 punch from 0x089e49 to 0x08bc5b (0x1e13 bytes) 000317[0] 1788801702.048370 mapwrite 0x6be871 thru 0x6bfd7d (0x150d bytes) 000318[0] 1788801702.150982 trunc from 0x7651f0 to 0x41d010 (0x3481df bytes) 000319[1] 1788801702.251568 trunc from 0x41d011 to 0x295f32 (0x1870de bytes) 000320[1] 1788801702.302810 trunc from 0x295f33 to 0x609a63 (0x373b31 bytes) 000321[0] 1788801702.348514 write 0x4fc060 thru 0x4fc4c5 (0x466 bytes) 000322[0] 1788801702.380350 trunc from 0x609a64 to 0x65b052 (0x515ef bytes) 000324[0] 1788801702.419781 mapwrite 0x355835 thru 0x355a67 (0x233 bytes) 000326[1] 1788801702.550339 write 0x4a366a thru 0x4a50f6 (0x1a8d bytes) 000327[1] 1788801702.620701 trunc from 0x65b053 to 0x5fa9c5 (0x6068d bytes) 000330[1] 1788801702.785855 mapread 0x3a66d2 thru 0x3a85af (0x1ede bytes) 000333[1] 1788801702.845188 trunc from 0x5fa9c6 to 0x03bc9d (0x5bed28 bytes) 000334[0] 1788801702.919145 read 0x018998 thru 0x019b70 (0x11d9 bytes) 000339[1] 1788801703.028558 mapwrite 0x3ad2f4 thru 0x3ae515 (0x1222 bytes) 000340[0] 1788801703.124390 punch from 0x2da086 to 0x2dba04 (0x197f bytes) 000341[0] 1788801703.159893 mapwrite 0x7e5bf4 thru 0x7e60d2 (0x4df bytes) 000344[0] 1788801703.472589 write 0x242d38 thru 0x243927 (0xbf0 bytes) 000347[1] 1788801703.513304 mapwrite 0x828094 thru 0x829b0d (0x1a7a bytes) 000350[1] 1788801703.717101 mapwrite 0x490c2e thru 0x492587 (0x195a bytes) 000356[0] 1788801703.970172 mapread 0x29a5f2 thru 0x29b0b6 (0xac5 bytes) 000358[1] 1788801704.017865 write 0x61f0b2 thru 0x621053 (0x1fa2 bytes) 000359[0] 1788801704.042882 punch from 0x7a9a25 to 0x7a9bc7 (0x1a3 bytes) 000360[1] 1788801704.085871 write 0x410b81 thru 0x410ed4 (0x354 bytes) 000362[0] 1788801704.140163 read 0x2489f0 thru 0x24a7bd (0x1dce bytes) 000363[1] 1788801704.146578 punch from 0x2fe539 to 0x2fed2f (0x7f7 bytes) 000365[0] 1788801704.183744 trunc from 0x829b0e to 0x8b6642 (0x8cb35 bytes) 000367[1] 1788801704.231129 mapread 0x7dcf0f thru 0x7de573 (0x1665 bytes) 000368[0] 1788801704.316582 punch from 0x8531f5 to 0x853503 (0x30f bytes) 000369[1] 1788801704.329601 punch from 0x11d164 to 0x11d73d (0x5da bytes) 000370[1] 1788801704.355349 mapread 0x5b763e thru 0x5b7a0f (0x3d2 bytes) 000371[0] 1788801704.363883 trunc from 0x8b6643 to 0x8766ad (0x3ff95 bytes) 000372[1] 1788801704.418466 mapwrite 0x80bece thru 0x80c824 (0x957 bytes) 000373[1] 1788801704.534725 punch from 0x4bb76d to 0x4bbc7f (0x513 bytes) 000375[0] 1788801704.585275 read 0x2ad1b7 thru 0x2ae402 (0x124c bytes) 000378[0] 1788801704.685534 mapwrite 0x0cd1ba thru 0x0ce070 (0xeb7 bytes) 000379[1] 1788801704.808598 read 0x59330b thru 0x59333a (0x30 bytes) 000380[0] 1788801704.840067 mapwrite 0x4c1729 thru 0x4c1864 (0x13c bytes) 000381[0] 1788801704.953276 mapread 0x033a78 thru 0x034df6 (0x137f bytes) 000383[1] 1788801704.991709 mapread 0x33b108 thru 0x33d0af (0x1fa8 bytes) 000384[1] 1788801705.118715 read 0x3f8469 thru 0x3f87f5 (0x38d bytes) 000386[0] 1788801705.163342 mapread 0x82ab5f thru 0x82b2a1 (0x743 bytes) 000388[0] 1788801705.193081 mapwrite 0x9ae1e1 thru 0x9aee46 (0xc66 bytes) 000389[0] 1788801705.305690 punch from 0x58c309 to 0x58d017 (0xd0f bytes) 000390[0] 1788801705.334379 write 0x0d9518 thru 0x0dacd1 (0x17ba bytes) 000393[0] 1788801705.403266 trunc from 0x9aee47 to 0x42e792 (0x5806b4 bytes) 000394[0] 1788801705.452218 punch from 0x34d2af to 0x34db97 (0x8e9 bytes) 000395[0] 1788801705.459588 mapwrite 0x18418e thru 0x1845cc (0x43f bytes) 000396[0] 1788801705.523531 trunc from 0x42e793 to 0x576c4c (0x1484ba bytes) truncating to largest ever: 0x99537e 000397[1] 1788801705.563432 trunc from 0x576c4d to 0x99537d (0x41e731 bytes) 000398[1] 1788801705.617511 write 0x96e552 thru 0x970065 (0x1b14 bytes) 000399[1] 1788801705.657632 mapread 0x4d52f6 thru 0x4d702e (0x1d39 bytes) 000400[1] 1788801705.684054 mapread 0x0fe8ef thru 0x0ffd62 (0x1474 bytes) 000401[1] 1788801705.734425 read 0x44bb6e thru 0x44d7d4 (0x1c67 bytes) 000402[0] 1788801705.740856 mapwrite 0x03acf6 thru 0x03b3e0 (0x6eb bytes) 000403[0] 1788801705.938106 write 0x5d3f19 thru 0x5d4793 (0x87b bytes) 000404[0] 1788801705.991571 mapread 0x3b5151 thru 0x3b66dc (0x158c bytes) 000405[0] 1788801706.006140 punch from 0x8b23e3 to 0x8b278f (0x3ad bytes) 000406[1] 1788801706.028853 punch from 0x406cba to 0x408bbe (0x1f05 bytes) 000407[0] 1788801706.072204 read 0x43d58f thru 0x43ee86 (0x18f8 bytes) 000408[0] 1788801706.089479 mapwrite 0x717d0b thru 0x717d1e (0x14 bytes) 000411[0] 1788801706.245117 mapread 0x5f560f thru 0x5f6a3d (0x142f bytes) 000414[0] 1788801706.302508 mapwrite 0x6ab6d1 thru 0x6aba71 (0x3a1 bytes) 000415[1] 1788801706.449734 mapwrite 0x1dc4dd thru 0x1dd39a (0xebe bytes) 000416[1] 1788801706.618242 write 0x57c1e0 thru 0x57ca4c (0x86d bytes) 000417[0] 1788801706.668776 punch from 0x8dc225 to 0x8dc9f0 (0x7cc bytes) 000421[1] 1788801706.740225 write 0x8e9870 thru 0x8ea16a (0x8fb bytes) 000422[1] 1788801706.762793 mapread 0x1ac34e thru 0x1addbb (0x1a6e bytes) 000423[1] 1788801706.792181 trunc from 0x99537e to 0x8ec7ff (0xa8b7e bytes) 000426[0] 1788801706.876757 trunc from 0x8ec800 to 0x3360b1 (0x5b674e bytes) 000427[1] 1788801706.938773 trunc from 0x3360b2 to 0x1bb7a6 (0x17a90b bytes) 000429[0] 1788801706.995784 read 0x0e8fd5 thru 0x0ea052 (0x107e bytes) 000433[1] 1788801707.070275 read 0x072386 thru 0x072dd7 (0xa52 bytes) 000434[1] 1788801707.114811 trunc from 0x1bb7a7 to 0x78ce3f (0x5d1699 bytes) 000436[1] 1788801707.198384 punch from 0x1a5dc4 to 0x1a72e4 (0x1521 bytes) 000438[1] 1788801707.224485 write 0x92f64c thru 0x930baf (0x1564 bytes) 000442[1] 1788801707.262236 write 0x13d9a2 thru 0x13f096 (0x16f5 bytes) 000443[0] 1788801707.353632 mapwrite 0x25676a thru 0x25793a (0x11d1 bytes) 000444[0] 1788801707.499202 write 0x96d09d thru 0x96d32d (0x291 bytes) 000445[0] 1788801707.521577 mapread 0x5903f0 thru 0x59235b (0x1f6c bytes) 000447[1] 1788801707.643713 write 0x486b38 thru 0x48700a (0x4d3 bytes) 000449[1] 1788801707.765616 write 0x433107 thru 0x433e78 (0xd72 bytes) 000450[0] 1788801707.815350 mapwrite 0x32ed42 thru 0x32fd27 (0xfe6 bytes) 000451[0] 1788801707.908041 read 0x8295b2 thru 0x82a816 (0x1265 bytes) 000453[0] 1788801707.950073 write 0x1576b2 thru 0x15855b (0xeaa bytes) 000457[1] 1788801708.071567 trunc from 0x96d32e to 0x4e8a5c (0x4848d1 bytes) 000460[0] 1788801708.188608 trunc from 0x4e8a5d to 0x066f4e (0x481b0e bytes) 000461[0] 1788801708.280256 read 0x02d109 thru 0x02d176 (0x6e bytes) 000462[1] 1788801708.321190 read 0x05a6a4 thru 0x05b9d6 (0x1333 bytes) 000463[1] 1788801708.360564 mapwrite 0x62b1ce thru 0x62bd4a (0xb7d bytes) 000464[1] 1788801708.561189 mapread 0x238e7c thru 0x239a09 (0xb8e bytes) 000465[0] 1788801708.582273 trunc from 0x62bd4b to 0x97569a (0x349950 bytes) 000467[0] 1788801708.630212 trunc from 0x97569b to 0x4194ef (0x55c1ab bytes) 000468[0] 1788801708.660771 trunc from 0x4194f0 to 0x1736da (0x2a5e15 bytes) 000469[0] 1788801708.717575 mapwrite 0x92ef9f thru 0x930834 (0x1896 bytes) 000470[0] 1788801709.067614 punch from 0x37ae1d to 0x37bd72 (0xf56 bytes) 000471[0] 1788801709.098385 read 0x77464f thru 0x775d7f (0x1731 bytes) 000472[1] 1788801709.123293 write 0x573d9b thru 0x575901 (0x1b67 bytes) 000473[1] 1788801709.258162 punch from 0x37b416 to 0x37c060 (0xc4b bytes) 000474[1] 1788801709.296779 mapwrite 0x4275a5 thru 0x429180 (0x1bdc bytes) 000475[0] 1788801709.444872 mapread 0x340f18 thru 0x3416be (0x7a7 bytes) 000477[1] 1788801709.496294 mapread 0x4deb5a thru 0x4df26c (0x713 bytes) 000480[1] 1788801709.524561 mapwrite 0x2c04ba thru 0x2c07f8 (0x33f bytes) 000483[1] 1788801709.618559 mapwrite 0x0657fb thru 0x06702f (0x1835 bytes) 000486[1] 1788801709.771557 trunc from 0x930835 to 0x89e8d9 (0x91f5b bytes) 000488[1] 1788801709.817490 read 0x5bd7c7 thru 0x5befac (0x17e6 bytes) 000489[0] 1788801709.818274 read 0x1131f0 thru 0x11417f (0xf90 bytes) 000491[1] 1788801709.896159 write 0x1fad2c thru 0x1fae78 (0x14d bytes) 000492[0] 1788801709.921635 read 0x83c0d5 thru 0x83e095 (0x1fc1 bytes) 000494[0] 1788801709.993313 punch from 0x7954ad to 0x7962c9 (0xe1d bytes) 000497[1] 1788801710.048775 mapread 0x088121 thru 0x0884f4 (0x3d4 bytes) 000498[0] 1788801710.060649 write 0x3abb11 thru 0x3aca73 (0xf63 bytes) 000499[0] 1788801710.108256 punch from 0x3340b0 to 0x334312 (0x263 bytes) 000500[1] 1788801710.138153 mapwrite 0x3f8992 thru 0x3f8d7d (0x3ec bytes) All operations completed A-OK! PASS 16b (31s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 13:21:54 (1788801714) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 13:21:58 (1788801718) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.49918 s, 16.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.29723 s, 28.2 MB/s ldlm.namespaces.MGC192.168.206.153@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff942d82f2e800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff942d8601d800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff942d82f2e800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff942d82f2e800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4084 s, 10.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.8947 s, 36.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.20062 s, 47.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.48746 s, 19.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.26498 s, 12.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.73607 s, 28.1 MB/s PASS 16d (44s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 13:22:43 (1788801763) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.347173 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167641 s, 6.3 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 13:22:50 (1788801770) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 13:23:17 (1788801797) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 13:23:45 (1788801825) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (11s) == sanityn test 16i: read after truncate file ============ 13:23:56 (1788801836) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (10s) == sanityn test 16j: race dio with buffered i/o ========== 13:24:06 (1788801846) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.33934 s, 19.6 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.77595 s, 14.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.647 s, 7.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.37937 s, 19.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.06329 s, 12.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.02593 s, 12.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.47025 s, 10.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.33277 s, 19.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.81884 s, 14.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.97721 s, 8.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.07449 s, 8.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.19724 s, 11.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.38537 s, 11.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.19227 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.29124 s, 8.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.5481 s, 10.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.63237 s, 10.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.29122 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.36788 s, 7.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (38s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 13:24:45 (1788801885) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: : Operation not supported Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Sep 7 13:25:28 EDT 2026 + fsx 2 done: rc=0 Mon Sep 7 13:25:28 EDT 2026 + fsx 3 done: rc=0 Mon Sep 7 13:25:28 EDT 2026 + fsx 4 done: rc=0 Mon Sep 7 13:25:28 EDT 2026 + fsx 5 done: rc=0 Mon Sep 7 13:25:28 EDT 2026 PASS 16k (52s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 13:25:36 (1788801936) fail_loc=0x8000030a PASS 17 (9s) == sanityn test 18: mmap sanity check =========================================================================================== 13:25:45 (1788801945) mmap test1: basic mmap operation (PASS, 0.10517s) mmap test2: MAP_PRIVATE not write back (PASS, 0.063082s) mmap test3: concurrent mmap ops on two nodes (PASS, 14.787s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4386s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.13883s) mmap test6: check mmap write/read content on two nodes (PASS, 0.18513s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.119s) mmap test11: multi-thread shared mmap (PASS, 1.2562s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1229s) PASS 18 (30s) == sanityn test 19: test concurrent uncached read races ========================================================================= 13:26:15 (1788801975) oleg653-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg653-client: oleg653-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 13:26:19 (1788801979) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 13:26:26 (1788801986) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 13:26:31 (1788801991) now is 1788801994 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 new atime is 1788802055 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 13:27:41 (1788802061) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 13:27:47 (1788802067) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 13:27:54 (1788802074) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 13:28:01 (1788802081) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 13:28:04 (1788802084) utime: good mknod times 1788802084 <= 1788802085 <= 1788802085 for /mnt/lustre/f26a utime: good mknod times 1788802084 <= 1788802085 <= 1788802085 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 13:28:10 (1788802090) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 13:28:19 (1788802099) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-07 13:28:20.000000000 -0400 Birth: 2026-09-07 13:28:21.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-07 13:28:20.000000000 -0400 Birth: 2026-09-07 13:28:21.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 13:28:26 (1788802106) 0 PASS 30 (7s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 13:28:34 (1788802114) fail_loc=0x314 PASS 31a (6s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 13:28:41 (1788802121) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg653-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.153@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (20s) == sanityn test 31r: open-rename(replace) race =========== 13:29:00 (1788802140) -rw-r--r-- 1 root root 0 Sep 7 13:29 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 13:29:09 (1788802149) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43158 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 13:29:15 (1788802155) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 13:29:22 (1788802162) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 13:29:25 (1788802165) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34817 s, 38.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.12652 s, 46.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.51791 s, 34.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.7971 s, 29.2 MB/s contention_seconds 4, contention_events 20 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.94375 s, 27.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.05311 s, 25.5 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.03237 s, 50.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.07524 s, 48.8 MB/s PASS 32c (24s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 13:29:50 (1788802190) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 13:29:53 (1788802193) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 13:29:55 (1788802195) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 13:29:58 (1788802198) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 13:30:00 (1788802200) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff942d82f2e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff942d82f2e800.ost_server_uuid in IDLE state after 0 sec oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff942d82f2e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff942d82f2e800.ost_server_uuid in FULL state after 0 sec oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff942d82f2e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff942d82f2e800.ost_server_uuid in IDLE state after 0 sec oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff942d82f2e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff942d82f2e800.ost_server_uuid in FULL state after 0 sec oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff942d82f2e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff942d82f2e800.ost_server_uuid in IDLE state after 0 sec oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff942d82f2e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff942d82f2e800.ost_server_uuid in FULL state after 0 sec PASS 34 (124s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 13:32:04 (1788802324) fail_loc=0x80000317 Race attempt 0 Wait for 50455 50514 for 60 sec... - open/close 723 (time 1788802340.93 total 10.01 last 72.24) - open/close 1469 (time 1788802350.93 total 20.01 last 74.59) - open/close 2207 (time 1788802360.93 total 30.01 last 73.79) - open/close 2841 (time 1788802370.94 total 40.02 last 63.36) - open/close 3427 (time 1788802380.94 total 50.02 last 58.58) total: 4000 open/close in 58.22 seconds: 68.71 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.153@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.206.153@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 13:33:16 (1788802396) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg653-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.42646 s, 36.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2848 s, 40.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7493632) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48982 s, 35.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7491584) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.15084 s, 45.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.09743 s, 47.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.58318 s, 33.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.61998 s, 32.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.38514 s, 37.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.27867 s, 41.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7493632) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.77761 s, 29.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.52697 s, 20.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7446528) after(7497728) PASS 36 (317s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 13:38:32 (1788802712) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 - create 1345 (time 1788802726.51 total 10.01 last 134.40) - create 2722 (time 1788802736.52 total 20.02 last 137.59) - create 4061 (time 1788802746.52 total 30.02 last 133.87) - create 5392 (time 1788802756.52 total 40.02 last 133.07) - create 6783 (time 1788802766.52 total 50.02 last 139.04) - create 7992 (time 1788802776.53 total 60.03 last 120.85) - create 9399 (time 1788802786.53 total 70.03 last 140.69) total: 10000 create in 74.42 seconds: 134.38 ops/second PASS 37 (85s) == sanityn test 39a: file mtime does not change after rename ========================================================== 13:39:57 (1788802797) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 13:40:04 (1788802804) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 13:40:12 (1788802812) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 13:40:21 (1788802821) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 13:40:29 (1788802829) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 13:40:47 (1788802847) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (19s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 13:41:05 (1788802865) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (16s) == sanityn test 40e: pdirops: rename and others ======================================================================== 13:41:22 (1788802882) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (14s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 13:41:37 (1788802897) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (11s) == sanityn test 41b: pdirops: create vs create ======================================================================== 13:41:49 (1788802909) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 13:42:01 (1788802921) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (13s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 13:42:14 (1788802934) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 13:42:27 (1788802947) Conflict PASS 41e (12s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 13:42:39 (1788802959) Conflict PASS 41f (11s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 13:42:50 (1788802970) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 13:43:02 (1788802982) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (12s) == sanityn test 41i: reint_open: create vs create ======== 13:43:14 (1788802994) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1027s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 14:00:21 (1788804021) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 14:00:35 (1788804035) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 14:00:49 (1788804049) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (16s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 14:01:04 (1788804064) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 14:01:17 (1788804077) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (12s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 14:01:29 (1788804089) Conflict PASS 42f (13s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 14:01:43 (1788804103) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 14:01:56 (1788804116) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 14:02:09 (1788804129) PASS 43a (78s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 14:03:27 (1788804207) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 14:03:41 (1788804221) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 14:03:55 (1788804235) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 14:04:08 (1788804248) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 14:04:22 (1788804262) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 14:04:35 (1788804275) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (15s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 14:04:51 (1788804291) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 14:05:05 (1788804305) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 14:05:08 (1788804308) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (132s) == sanityn test 43k: unlink vs create ==================== 14:07:20 (1788804440) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1010s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 14:24:10 (1788805450) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Sep 7 14:24:15 EDT 2026 PASS 44a (8s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 14:24:19 (1788805459) open: File exists Conflict PASS 44b (9s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 14:24:27 (1788805467) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (10s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 14:24:37 (1788805477) Conflict PASS 44d (10s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 14:24:47 (1788805487) Conflict PASS 44e (9s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 14:24:56 (1788805496) Conflict PASS 44f (9s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 14:25:05 (1788805505) Conflict PASS 44g (9s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 14:25:14 (1788805514) Conflict PASS 44h (9s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 14:25:24 (1788805524) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (2s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 14:25:26 (1788805526) PASS 45a (96s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 14:27:02 (1788805622) Conflict PASS 45b (10s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 14:27:12 (1788805632) Conflict PASS 45c (9s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 14:27:22 (1788805642) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (8s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 14:27:30 (1788805650) Conflict PASS 45e (9s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 14:27:39 (1788805659) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (9s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 14:27:48 (1788805668) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (9s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 14:27:57 (1788805677) Conflict PASS 45h (9s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 14:28:06 (1788805686) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (2s) == sanityn test 45j: read vs rename ====================== 14:28:08 (1788805688) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (655s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 14:39:03 (1788806343) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (7s) == sanityn test 46b: pdirops: link vs create ======================================================================== 14:39:10 (1788806350) open: File exists Conflict PASS 46b (7s) == sanityn test 46c: pdirops: link vs link ======================================================================== 14:39:17 (1788806357) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (7s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 14:39:24 (1788806364) Conflict PASS 46d (7s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 14:39:31 (1788806371) Conflict PASS 46e (7s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 14:39:38 (1788806378) Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 14:39:45 (1788806385) Conflict PASS 46g (8s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 14:39:52 (1788806392) Conflict PASS 46h (7s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 14:39:59 (1788806399) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (2s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 14:40:01 (1788806401) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 14:40:02 (1788806402) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (2s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 14:40:04 (1788806404) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 14:40:06 (1788806406) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 14:40:07 (1788806407) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 14:40:09 (1788806409) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 14:40:11 (1788806411) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 14:40:12 (1788806412) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.073337 s, 140 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (8s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 14:40:21 (1788806421) 0+1 records in 0+1 records out 158 bytes copied, 0.00739592 s, 21.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218444) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 14:40:26 (1788806426) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00742706 s, 138 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336499 s, 31.2 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 14:40:41 (1788806441) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0163105 s, 62.8 kB/s PASS 51c (8s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 14:40:49 (1788806449) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254742 s, 41.2 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 14:40:54 (1788806454) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (5s) == sanityn test 54: rename locking ======================= 14:40:59 (1788806459) llite.lustre-ffff942d82f2e800.xattr_cache=1 llite.lustre-ffff942d8601d800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 14:41:00.000000000 -0400 Modify: 2026-09-07 14:41:00.000000000 -0400 Change: 2026-09-07 14:41:00.000000000 -0400 Birth: 2026-09-07 14:41:00.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 14:41:05.000000000 -0400 Modify: 2026-09-07 14:41:06.000000000 -0400 Change: 2026-09-07 14:41:06.000000000 -0400 Birth: 2026-09-07 14:41:06.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 14:41:11.000000000 -0400 Modify: 2026-09-07 14:41:11.000000000 -0400 Change: 2026-09-07 14:41:11.000000000 -0400 Birth: 2026-09-07 14:41:11.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 14:41:11.000000000 -0400 Modify: 2026-09-07 14:41:11.000000000 -0400 Change: 2026-09-07 14:41:11.000000000 -0400 Birth: 2026-09-07 14:41:11.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 14:41:16.000000000 -0400 Modify: 2026-09-07 14:41:17.000000000 -0400 Change: 2026-09-07 14:41:17.000000000 -0400 Birth: 2026-09-07 14:41:17.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 14:41:16.000000000 -0400 Modify: 2026-09-07 14:41:17.000000000 -0400 Change: 2026-09-07 14:41:17.000000000 -0400 Birth: 2026-09-07 14:41:17.000000000 -0400 PASS 54 (25s) == sanityn test 55a: rename vs unlink target dir ========= 14:41:24 (1788806484) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (9s) == sanityn test 55b: rename vs unlink source dir ========= 14:41:33 (1788806493) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (9s) == sanityn test 55c: rename vs unlink orphan target dir == 14:41:42 (1788806502) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (14s) == sanityn test 55d: rename file vs link ================= 14:41:56 (1788806516) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 14:42:06 (1788806526) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 14:42:08 (1788806528) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (14s) == sanityn test 55g: rename: race with trylock =========== 14:42:22 (1788806542) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (15s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 14:42:37 (1788806557) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788806558 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 1.35642 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (49s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 14:43:26 (1788806606) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788806608 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.13011 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (44s) == sanityn test 60: Verify data_version behaviour ======== 14:44:10 (1788806650) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 14:44:13 (1788806653) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 14:44:16 (1788806656) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 14:44:19 (1788806659) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00401683 s, 16.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 14:44:21 (1788806661) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00503008 s, 13.0 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 14:44:22 (1788806662) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 14:44:24 (1788806664) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00624063 s, 10.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 14:44:25 (1788806665) llite.lustre-ffff942d82f2e800.xattr_cache=1 llite.lustre-ffff942d8601d800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (3s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 14:44:28 (1788806668) llite.lustre-ffff942d82f2e800.xattr_cache=1 llite.lustre-ffff942d8601d800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 14:44:31 (1788806671) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00544836 s, 188 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00764844 s, 134 kB/s 231213: get lock2 231214: get lock1 231214: done 231214: get lock2 231213: done 231213: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231214: cannot get lock2: Resource deadlock avoided 231214: exit rc=0 fcntl cmd 7 failed: Resource deadlock avoided 231213: cannot get lock1: Resource deadlock avoided 231213: done 231213: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 14:44:37 (1788806677) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0224838 s, 93.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018432 s, 56.9 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231795 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 14:44:44 (1788806684) opening files: ........ closing files: ........ PASS 76 (26s) == sanityn test 77a: check FIFO NRS policy =============== 14:45:11 (1788806711) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.174147 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399228 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348853 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983121 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154308 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125534 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162961 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162266 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17038 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160281 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199181 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188444 s, 5.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203217 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195688 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193045 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19895 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212227 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00795333 s, 132 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458421 s, 229 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367116 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176152 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252508 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175709 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339415 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00224008 s, 468 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453859 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413482 s, 25.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490893 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270402 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150356 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137045 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00840167 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122265 s, 85.8 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 14:45:14 (1788806714) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155025 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418405 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619495 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806129 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730185 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851719 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970805 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103426 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850012 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953348 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914819 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106244 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103635 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779229 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109164 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905825 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103575 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145668 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167784 s, 62.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210137 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222342 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193083 s, 54.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206335 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242781 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150069 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00646324 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214466 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00375026 s, 280 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208815 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306461 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277099 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177637 s, 59.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00927641 s, 113 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145918 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294263 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480731 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548614 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864867 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760235 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832489 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110386 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103591 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975248 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107194 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105074 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935366 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10165 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101254 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916319 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994788 s, 10.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157655 s, 66.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163453 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199639 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161186 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168957 s, 62.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231663 s, 45.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023317 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163501 s, 64.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216965 s, 48.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00920091 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025096 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0069481 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123126 s, 85.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116314 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00779478 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458249 s, 229 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 14:45:18 (1788806718) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.187704 s, 89.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017235 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450508 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577696 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465085 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660233 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512209 s, 20.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674497 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108917 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860169 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108132 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10577 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117612 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105691 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110028 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129792 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133157 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00986532 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214076 s, 49.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165694 s, 63.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138653 s, 75.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012011 s, 87.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267964 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00323369 s, 324 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228088 s, 46.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826675 s, 127 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373117 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210423 s, 49.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205301 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219084 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00643048 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232709 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00357932 s, 293 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.166693 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562746 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103174 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108703 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104111 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954523 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119699 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143702 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115748 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143314 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135895 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146841 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10869 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13009 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141363 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149622 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123774 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281305 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230628 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169775 s, 61.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200142 s, 52.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286742 s, 36.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198532 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017479 s, 60.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323369 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372814 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254945 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254069 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304144 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343111 s, 30.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263615 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325745 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00515762 s, 203 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.161453 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448027 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08267 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932432 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116364 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121852 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148679 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131096 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136483 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123159 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161847 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113521 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149247 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125938 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15782 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155542 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170114 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189858 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246612 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205455 s, 51.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144092 s, 72.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190394 s, 55.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00633514 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191012 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042679 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00440758 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449243 s, 233 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038182 s, 275 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00352987 s, 297 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00871922 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171803 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0027482 s, 382 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00329244 s, 318 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 14:45:24 (1788806724) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.230075 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220158 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300425 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540687 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715186 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872584 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596056 s, 17.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644751 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886811 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886162 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840739 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968995 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878873 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999257 s, 10.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232382 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126152 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283794 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107083 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224244 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250363 s, 41.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337582 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016898 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236519 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108455 s, 96.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183317 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250823 s, 41.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00818706 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270499 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288613 s, 36.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311985 s, 33.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122333 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333703 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394627 s, 266 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.168405 s, 99.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282559 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040303 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468425 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820497 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761758 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785911 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112622 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106949 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105064 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113903 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962216 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103255 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092193 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994584 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120196 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122487 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164603 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237993 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102777 s, 102 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214284 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124568 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136069 s, 77.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118479 s, 88.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319671 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219848 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017718 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210901 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00364793 s, 287 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289103 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02967 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00426463 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309143 s, 33.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143367 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038655 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308608 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479796 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550085 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736439 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087708 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789408 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669476 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086905 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110854 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108108 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867384 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109063 s, 9.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117635 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116048 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127903 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228829 s, 45.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224876 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209319 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233994 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188225 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173611 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280519 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00545073 s, 192 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283728 s, 37.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253472 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208958 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206069 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229863 s, 45.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211226 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00949729 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317063 s, 33.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (6s) == sanityn test 77e: check TBF NID nrs policy ============ 14:45:30 (1788806730) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.53@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.146976 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255197 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261635 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321682 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890218 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846391 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112344 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999665 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884144 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100934 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118481 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872506 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103955 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938066 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126071 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123559 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1142 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171843 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175148 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178096 s, 58.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142009 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224403 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313799 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188505 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211752 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253204 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251553 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226122 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240084 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298036 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00505878 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275941 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274936 s, 38.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.199913 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265707 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453594 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771268 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653896 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981143 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915749 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866948 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932139 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902875 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105678 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1129 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997364 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943694 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948233 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108134 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104107 s, 10.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191276 s, 5.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182851 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191862 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185141 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194608 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184332 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199054 s, 5.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206357 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202192 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197045 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201371 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198032 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206996 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19306 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18279 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197586 s, 5.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155929 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534602 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105748 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117904 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125729 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121762 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122293 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150519 s, 7.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132617 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116554 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129519 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141645 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157143 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169846 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13275 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178956 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154603 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180643 s, 58.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247101 s, 42.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194235 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166261 s, 63.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192331 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194154 s, 54.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0073116 s, 143 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129493 s, 81.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00987233 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285136 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00719076 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00344608 s, 304 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305921 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218681 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00391125 s, 268 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570045 s, 184 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.16159 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552269 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079638 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859498 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082059 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874282 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116903 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124011 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116664 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126516 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134871 s, 7.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146331 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164261 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151598 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117013 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163042 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167254 s, 6.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119182 s, 88.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297735 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186836 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351755 s, 29.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265682 s, 39.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316328 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285024 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270928 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039076 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452962 s, 23.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372628 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340661 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343152 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441782 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351041 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037854 s, 27.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (9s) == sanityn test 77f: check TBF JobID nrs policy ========== 14:45:39 (1788806739) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.230139 s, 72.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166284 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328285 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640903 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672878 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729315 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621082 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588961 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813383 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822439 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100703 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899158 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083588 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924766 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110697 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091312 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101684 s, 10.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173763 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00931292 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222164 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178086 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022395 s, 46.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173097 s, 60.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209251 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00952364 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166153 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174528 s, 60.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212862 s, 49.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242619 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226779 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00312365 s, 336 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367679 s, 285 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00479698 s, 219 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145921 s, 115 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461496 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764785 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717055 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913757 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958357 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805483 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919678 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856236 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107663 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865787 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109609 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901118 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105119 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12107 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114496 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111363 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190709 s, 55.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314383 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301818 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276287 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224782 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273713 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241698 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299484 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263001 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419131 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225696 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00489595 s, 214 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275484 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276384 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294294 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103104 s, 102 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.208458 s, 80.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411277 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368711 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134821 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133433 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157834 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174742 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183504 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174202 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176379 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182461 s, 5.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168869 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204452 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200766 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17937 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1863 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212338 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195846 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271952 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145576 s, 72.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178079 s, 58.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267669 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181735 s, 57.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177833 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187948 s, 55.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249033 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200318 s, 52.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221759 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243207 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269824 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280102 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241718 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303845 s, 34.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.20292 s, 82.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322999 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273506 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106587 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100381 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129436 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133807 s, 7.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14527 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145764 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139949 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148954 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135582 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160889 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140714 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157348 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162479 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d8601d800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff942d8601d800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157325 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291247 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297094 s, 35.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292113 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022419 s, 46.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026904 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304231 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377414 s, 27.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322878 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330577 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330199 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349082 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256251 s, 40.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307701 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332933 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380292 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394609 s, 26.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (8s) == sanityn test 77g: Change TBF type directly ============ 14:45:48 (1788806748) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 14:45:52 (1788806752) oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg653-client: oleg653-server: ssh exited with exit code 2 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 14:45:56 (1788806756) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (9s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 14:46:05 (1788806765) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.146493 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256552 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329233 s, 31.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951248 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09183 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794673 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804832 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792957 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845236 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102768 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117368 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103408 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899965 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11082 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967133 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970529 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115999 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112982 s, 92.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128486 s, 81.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227704 s, 46.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198061 s, 52.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181051 s, 57.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203134 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252069 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197897 s, 53.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447454 s, 234 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014267 s, 73.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138159 s, 75.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315475 s, 33.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00470861 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123979 s, 84.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318657 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00941488 s, 111 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96893 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4584 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (53s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 14:46:58 (1788806818) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23977 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644491 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828525 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720227 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646312 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943428 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839885 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860177 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546116 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141779 s, 7.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137553 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12656 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137274 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162048 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157805 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159936 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168395 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213216 s, 49.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018507 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185788 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251659 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255028 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252789 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294474 s, 35.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217021 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301973 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204164 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021185 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284727 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211244 s, 49.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263709 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280792 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320459 s, 32.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7094 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9463 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23178 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369593 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702804 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116227 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992361 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962623 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109402 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129557 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10131 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163155 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164857 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194452 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220033 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218398 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221161 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221494 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207616 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217275 s, 48.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297295 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266195 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321594 s, 32.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294805 s, 35.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354903 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338665 s, 31.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261073 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342803 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280116 s, 37.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031603 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316965 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342956 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373823 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328582 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350889 s, 29.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7432 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9093 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (135s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 14:49:13 (1788806953) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23164 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023359 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778645 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732286 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10222 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972436 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085791 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892528 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096274 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154076 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154127 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158699 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192586 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201783 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19688 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184377 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194372 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023014 s, 45.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283914 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298016 s, 35.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028219 s, 37.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278235 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321834 s, 32.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263921 s, 39.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293896 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324543 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382184 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336293 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337922 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028458 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00663636 s, 158 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408665 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100142 s, 105 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8423 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9677 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23046 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249849 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376908 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720705 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07292 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741822 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736182 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763389 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597523 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156017 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156135 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174018 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185684 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178863 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202796 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192844 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18729 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191913 s, 54.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259591 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290028 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227864 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259598 s, 40.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323352 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324854 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295207 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026337 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032192 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317674 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325233 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305879 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332211 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028594 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351435 s, 29.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7003 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9444 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (137s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 14:51:30 (1788807090) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.315399 s, 53.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035458 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329827 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220029 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125326 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124219 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126171 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148237 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146302 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161551 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1676 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156561 s, 6.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168828 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16361 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15524 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151128 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143207 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200038 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185764 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183402 s, 57.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200527 s, 52.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255877 s, 41.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242576 s, 43.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345129 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290521 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278641 s, 37.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031048 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222209 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293469 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279244 s, 37.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295341 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263948 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298401 s, 35.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98075 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77946 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.53@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.53@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.247119 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341649 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234591 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152861 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137455 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137269 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147641 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136049 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148393 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163227 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169572 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162298 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153147 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253003 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248807 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289261 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295949 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307002 s, 34.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244089 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250702 s, 41.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343631 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364011 s, 28.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182421 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104683 s, 100 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320023 s, 32.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520104 s, 202 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297759 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286751 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113569 s, 92.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389178 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360798 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00532731 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192151 s, 54.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9546 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76324 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.53@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.254293 s, 66.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285349 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294726 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670406 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101815 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930293 s, 11.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875472 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104094 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952674 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12367 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109957 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108417 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122004 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123968 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117525 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111087 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111374 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307081 s, 34.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302541 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319486 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274934 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027931 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028266 s, 37.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308323 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310783 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348126 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359596 s, 29.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343136 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031682 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359816 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393997 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413701 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448136 s, 23.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97622 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91736 s, 21.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.450179 s, 37.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256031 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050961 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716322 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800951 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669548 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716789 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749884 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108806 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938938 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119007 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921308 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090478 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245465 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241362 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290473 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307174 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104631 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245441 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00894671 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249864 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014366 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260357 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209848 s, 50.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214057 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023702 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246984 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259738 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272179 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272991 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274018 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288771 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297158 s, 35.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81787 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73602 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94673 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75658 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.85942 s, 19.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277974 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443293 s, 23.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443654 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402018 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582785 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297227 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440663 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435285 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156685 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179066 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171237 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194467 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209375 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188358 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207219 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20735 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535586 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218343 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181867 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251134 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231383 s, 45.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215701 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102407 s, 102 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274507 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018796 s, 55.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00811231 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00778973 s, 135 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286773 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00963092 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00260401 s, 403 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00238604 s, 439 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.689 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.444 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.248285 s, 67.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389506 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391827 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147767 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124672 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123369 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13506 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135579 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14669 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134608 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144125 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129909 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151676 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155706 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166083 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14003 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153224 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158322 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192741 s, 54.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225949 s, 46.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173103 s, 60.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175242 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189567 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155852 s, 67.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187452 s, 55.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00476934 s, 220 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231313 s, 45.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219347 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123568 s, 84.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285828 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00504842 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460533 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00445403 s, 235 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93997 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74801 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.444359 s, 37.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02704 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356758 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616236 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058161 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558091 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089078 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977092 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091237 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103869 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855809 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933181 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933576 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320777 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307097 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35455 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372366 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169975 s, 61.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244908 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229625 s, 45.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215394 s, 48.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211063 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259208 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232507 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299125 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310721 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157555 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00854229 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147262 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00445515 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148203 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383949 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0060873 s, 172 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80362 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7366 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77678 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7393 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (355s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 14:57:25 (1788807445) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.260287 s, 64.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246255 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264405 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720976 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700396 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748241 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702202 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562988 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807437 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063798 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786489 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695422 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811088 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686493 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901033 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973275 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107066 s, 9.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191607 s, 54.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126115 s, 83.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015183 s, 69.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208921 s, 50.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299742 s, 35.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00759498 s, 138 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341003 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266335 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01175 s, 89.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145869 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105137 s, 99.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119011 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162874 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00857107 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281137 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218495 s, 48.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95174 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73291 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (36s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 14:58:01 (1788807481) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.53@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.53@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.23948 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298311 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254278 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118371 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113958 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991055 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11795 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136625 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117873 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125726 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14056 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140944 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122725 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255153 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245348 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301593 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295511 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016132 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151618 s, 69.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130165 s, 80.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186053 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014615 s, 71.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160689 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00954471 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213207 s, 49.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102535 s, 102 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0061788 s, 170 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218317 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221391 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234881 s, 44.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028739 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00346743 s, 302 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00516727 s, 203 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95115 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73452 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (35s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 14:58:36 (1788807516) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.53@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.238228 s, 70.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234801 s, 44.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293804 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508105 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480631 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383734 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043213 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791759 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668919 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917298 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927176 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791245 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10622 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871086 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10241 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10221 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100956 s, 10.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756562 s, 139 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120104 s, 87.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148453 s, 70.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130889 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214839 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011214 s, 93.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200439 s, 52.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216259 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011654 s, 90.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00879979 s, 119 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224398 s, 46.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00704869 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00392043 s, 267 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00970018 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00372592 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280035 s, 37.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92265 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96919 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (32s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 14:59:08 (1788807548) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.455064 s, 36.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552711 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534113 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563329 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828893 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878178 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103649 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984885 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119543 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131917 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105294 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104132 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111911 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130969 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161046 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310291 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303392 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194525 s, 53.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223497 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108116 s, 97.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304134 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117742 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116563 s, 90.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208812 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00620858 s, 169 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190929 s, 54.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261197 s, 40.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273717 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0046705 s, 225 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00466063 s, 225 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108941 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118471 s, 88.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393533 s, 266 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79565 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73476 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94869 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73284 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (81s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 15:00:29 (1788807629) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.851581 s, 19.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353283 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327341 s, 32.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367189 s, 28.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399083 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567756 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425408 s, 24.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290124 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533173 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159641 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190989 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16154 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188771 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199311 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20409 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211239 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200493 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015211 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140293 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119053 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133781 s, 78.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173155 s, 60.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175989 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186297 s, 56.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166041 s, 63.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00674159 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171442 s, 61.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00381082 s, 275 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00879646 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00370529 s, 283 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00427207 s, 245 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270916 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0054485 s, 192 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6667 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9603 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (63s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 15:01:32 (1788807692) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.244337 s, 68.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362031 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360733 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112737 s, 9.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133951 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125476 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131388 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133675 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170524 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182785 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182602 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194771 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186208 s, 5.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168989 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181665 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172376 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181553 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103961 s, 101 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024943 s, 42.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250609 s, 41.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191265 s, 54.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287667 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276165 s, 38.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190064 s, 55.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321548 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192571 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209095 s, 50.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234925 s, 44.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238631 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291844 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296859 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261896 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276769 s, 37.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94112 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73139 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.458907 s, 36.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470155 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509027 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364829 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508108 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960164 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834469 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736361 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698676 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922935 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116903 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136813 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125711 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137468 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137343 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153081 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159147 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287105 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113808 s, 92.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042546 s, 246 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00410353 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00408801 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224359 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00423452 s, 248 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238082 s, 44.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00628376 s, 167 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00445012 s, 236 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360819 s, 291 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374384 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311598 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00295077 s, 355 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00635703 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00379746 s, 276 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79878 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73447 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76213 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73516 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (118s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 15:03:30 (1788807810) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg653-client.virtnet /mnt/lustre (opts:) Stopping client oleg653-client.virtnet /mnt/lustre opts: Stopping clients: oleg653-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg653-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg653-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg653-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg653-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg653-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.53,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg653-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.53,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg653-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.53,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69766/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.53,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg653-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing load_modules_local oleg653-server: kmemleak disabled oleg653-server: Loading modules from /home/green/git/lustre-release/lustre oleg653-server: detected 4 online CPUs by sysfs oleg653-server: MODOPTS_LIBCFS= oleg653-server: Force libcfs to create 2 CPU partitions oleg653-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg653-server: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg653-server: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg653-server: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre Starting client oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre Started clients oleg653-client.virtnet: 192.168.206.153@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 Starting client oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 Started clients oleg653-client.virtnet: 192.168.206.153@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff942d9011f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff942d9016d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff942d9011f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff942d9016d800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241566 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301282 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389222 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605987 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679286 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699035 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820296 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892286 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851843 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745686 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692216 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928742 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801061 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806454 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883281 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136172 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142942 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201328 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244865 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273869 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242218 s, 43.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275974 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280411 s, 37.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320324 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310912 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304669 s, 34.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316934 s, 33.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304045 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289806 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315606 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302968 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331356 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302886 s, 34.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93048 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96842 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.268229 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704286 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120547 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125447 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135143 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113913 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127585 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132691 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151806 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138624 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147607 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145922 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150066 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210452 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240914 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27987 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270025 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208547 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182291 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162482 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130115 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171164 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015562 s, 67.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221516 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231259 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254928 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206729 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291515 s, 36.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299219 s, 35.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263438 s, 39.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200128 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279251 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299559 s, 35.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93989 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73463 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (112s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 15:05:22 (1788807922) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (7s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 15:05:29 (1788807929) On MGS 192.168.206.153, active = nodemap.active=1 On MGS 192.168.206.153, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.248976 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535324 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100705 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121592 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117113 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127415 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151316 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12714 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119927 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125835 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151966 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239906 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239413 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286649 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297329 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338724 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325468 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178875 s, 58.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022507 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286453 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321838 s, 32.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263211 s, 39.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254106 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158903 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021407 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302094 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256414 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306766 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131341 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020202 s, 51.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187529 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160278 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00205164 s, 511 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92824 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97146 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.206.153, TBF.id = On MGS 192.168.206.153, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.149312 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452374 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603041 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576015 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050272 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059164 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756491 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731003 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670068 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896665 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777585 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750184 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733484 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985304 s, 10.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d9016d800.lock_unused_count=6 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911189 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665581 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939441 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142003 s, 73.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00936996 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168719 s, 62.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139513 s, 75.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144658 s, 72.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00344377 s, 304 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103194 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0079577 s, 132 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205011 s, 51.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0097368 s, 108 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246391 s, 42.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284456 s, 36.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123129 s, 85.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181411 s, 57.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570596 s, 184 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263884 s, 39.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff942d9016d800.lock_unused_count=1 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.48557 s, 70.6 MB/s Write runtime is 3 s, speed is 33.333333 IOPS write rate (33.333333) should exceed limit (20 * 1) Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.530443 s, 198 MB/s Read runtime is 1 s, speed is 100.000000 IOPS read rate (100.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 On MGS 192.168.206.153, active = nodemap.active=1 On MGS 192.168.206.153, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.245987 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213661 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231315 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504084 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04047 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539221 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449236 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782482 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765018 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875293 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943215 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763397 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098358 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977727 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110709 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258426 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25529 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148931 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144884 s, 72.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180891 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158443 s, 66.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014417 s, 72.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167716 s, 62.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157812 s, 66.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120641 s, 86.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705538 s, 149 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210561 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00340281 s, 308 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212021 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215837 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106451 s, 98.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191253 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0042977 s, 244 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.92121 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73115 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.206.153, TBF.id = On MGS 192.168.206.153, active = nodemap.active=0 PASS 77kj (114s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 15:07:23 (1788808043) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 15:07:26 (1788808046) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg653-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.67697 s, 285 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg653-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0064 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 15:08:17 (1788808097) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.12169 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023098 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308192 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444755 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605589 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952538 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110266 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866429 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111507 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986062 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113689 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139257 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137637 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134502 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127208 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138578 s, 7.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff942d9016d800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff942d9016d800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133612 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017164 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189132 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021306 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199626 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119015 s, 88.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00714664 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385312 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00996861 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570317 s, 184 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257825 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132978 s, 78.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224405 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280927 s, 37.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364891 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264651 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180305 s, 58.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94522 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96952 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.237145 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250291 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181949 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952041 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969512 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110736 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135736 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145811 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130012 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129331 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130523 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142601 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126236 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276422 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267566 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293992 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314564 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133582 s, 78.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020369 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219933 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233105 s, 45.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264821 s, 39.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140949 s, 74.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220981 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231772 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305849 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00435741 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221809 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273825 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252819 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254276 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250034 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276121 s, 38.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9211 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96793 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (66s) == sanityn test 77o: Changing rank should not panic ====== 15:09:23 (1788808163) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 15:09:27 (1788808167) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (39s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 15:10:06 (1788808206) mds.MDS.mdt.nrs_policies=tbf oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg653-client: oleg653-server: ssh exited with exit code 36 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (12s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 15:10:18 (1788808218) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 276451 kill md_thread xargs PASS 77r (42s) == sanityn test 77s: Check TBF LRU shrinker ============== 15:11:00 (1788808260) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (12s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 15:11:12 (1788808272) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 15:11:15 (1788808275) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826936334 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 15:11:16.000000000 -0400 Modify: 2026-09-07 15:11:16.000000000 -0400 Change: 2026-09-07 15:11:16.000000000 -0400 Birth: 2026-09-07 15:11:17.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 15:11:28 (1788808288) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 15:11:29 (1788808289) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 15:11:30 (1788808290) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 15:11:31 (1788808291) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 15:11:32 (1788808292) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 15:11:33 (1788808293) total: 2500 open/close in 4.82 seconds: 518.73 ops/second snapshot_time 1788808313.727219249 secs.nsecs start_time 1788808294.708575067 secs.nsecs elapsed_time 19.018644182 secs.nsecs open 2505 samples [usecs] 73 11650 920130 494056526 close 2504 samples [usecs] 52 1029 244605 27030867 mknod 2501 samples [usecs] 193 9778 819532 382423430 unlink 2500 samples [usecs] 303 11620 1191248 989519524 mkdir 2 samples [usecs] 970 996 1966 1932916 rmdir 2 samples [usecs] 2871 14172 17043 209088225 rename 2500 samples [usecs] 249 5582 2679633 3929517187 getattr 3760 samples [usecs] 6 1172 71035 6351849 parallel_rename_file 2500 samples [usecs] 249 5582 2679633 3929517187 crossdir_rename 2500 samples [usecs] 249 5582 2679633 3929517187 readpage 12 samples [usecs] 1775 10189 45728 257615248 PASS 81d (21s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 15:11:54 (1788808314) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7627 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7627 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 15:11:57 (1788808317) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 15:11:58 (1788808318) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 15:12:05 (1788808325) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788808326 Finishing test test1 at 1788808326 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 15:12:08 (1788808328) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 15:12:09 (1788808329) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 15:12:10 (1788808330) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 15:12:11 (1788808331) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6487 0x1957 0x240000400 1 6449 0x1931 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6450 0x1932 0x280000400 0 6488 0x1958 0x240000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 15:12:20 (1788808340) osc.lustre-OST0000-osc-ffff942d9011f000.idle_timeout=0 osc.lustre-OST0000-osc-ffff942d9016d800.idle_timeout=0 osc.lustre-OST0001-osc-ffff942d9011f000.idle_timeout=0 osc.lustre-OST0001-osc-ffff942d9016d800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0331718 s, 247 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09722 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 297739 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0151991 s, 269 kB/s osc.lustre-OST0000-osc-ffff942d9011f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff942d9016d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff942d9011f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff942d9016d800.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 15:12:31 (1788808351) llite.lustre-ffff942d9011f000.fast_read=0 llite.lustre-ffff942d9016d800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0194304 s, 108 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff942d9011f000.fast_read=1 llite.lustre-ffff942d9016d800.fast_read=1 PASS 95a (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 15:12:45 (1788808365) llite.lustre-ffff942d9011f000.fast_read=0 llite.lustre-ffff942d9016d800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00182179 s, 6.7 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff942d9011f000.fast_read=1 llite.lustre-ffff942d9016d800.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 15:12:54 (1788808374) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 15:12:55 (1788808375) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00132723 s, 3.1 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 15:12:57 (1788808377) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0389553 s, 53.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 15:12:59 (1788808379) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0376083 s, 55.8 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 15:13:01 (1788808381) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0014532 s, 8.3 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 15:13:03 (1788808383) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00141578 s, 2.9 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 15:13:05 (1788808385) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00150975 s, 2.7 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 15:13:07 (1788808387) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00151806 s, 2.7 MB/s PASS 101c (3s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 15:13:10 (1788808390) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 ef 11 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0x11ef:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (3s) == sanityn test 103: Test size correctness with lockahead ========================================================== 15:13:13 (1788808393) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788808393 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788808396 Finishing test test23 at 1788808396 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 15:13:20 (1788808400) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 15:13:21 (1788808401) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 15:13:44 (1788808424) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 15:13:45 (1788808425) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0040383 s, 254 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938870 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 15:13:45.000000000 -0400 Modify: 2026-09-07 15:13:45.000000000 -0400 Change: 2026-09-07 15:13:45.000000000 -0400 Birth: 2026-09-07 15:13:46.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938870 -rw-r--r-- 1 0 0 1788808426 1788808425 1788808425 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938870 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 15:13:45.000000000 -0400 Modify: 2026-09-07 15:13:45.000000000 -0400 Change: 2026-09-07 15:13:45.000000000 -0400 Birth: 2026-09-07 15:13:46.000000000 -0400 1788808425 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 15:13:47 (1788808427) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 15:13:49 (1788808429) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0969403 s, 108 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6457 0x1939 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 15:13:54 (1788808434) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.103264 s, 102 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6497 0x1961 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 15:14:02 (1788808442) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00168558 s, 5.9 MB/s llite.lustre-ffff942d9011f000.hybrid_io=0 llite.lustre-ffff942d9016d800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.21138 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 15:14:09 (1788808449) 192.168.206.153@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg653-client.virtnet /mnt/lustre (opts:) 192.168.206.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg653-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 192.168.206.153@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg653-client.virtnet /mnt/lustre (opts:) Stopping client oleg653-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing load_modules_local oleg653-server: kmemleak disabled oleg653-server: Loading modules from /home/green/git/lustre-release/lustre oleg653-server: detected 4 online CPUs by sysfs oleg653-server: MODOPTS_LIBCFS= oleg653-server: Force libcfs to create 2 CPU partitions oleg653-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 192.168.206.153@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg653-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg653-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing load_modules_local oleg653-server: kmemleak disabled oleg653-server: Loading modules from /home/green/git/lustre-release/lustre oleg653-server: detected 4 online CPUs by sysfs oleg653-server: MODOPTS_LIBCFS= oleg653-server: Force libcfs to create 2 CPU partitions oleg653-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 2 Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 192.168.206.153@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg653-client.virtnet /mnt/lustre (opts:) Stopping client oleg653-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing load_modules_local oleg653-server: kmemleak disabled oleg653-server: Loading modules from /home/green/git/lustre-release/lustre oleg653-server: detected 4 online CPUs by sysfs oleg653-server: MODOPTS_LIBCFS= oleg653-server: Force libcfs to create 2 CPU partitions oleg653-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 PASS 109 (35s) == sanityn test 110: do not grant another lock on resend ========================================================== 15:14:44 (1788808484) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-07 15:14:45.000000000 -0400 Modify: 2026-09-07 15:14:45.000000000 -0400 Change: 2026-09-07 15:14:45.000000000 -0400 Birth: 2026-09-07 15:14:45.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg653-client.virtnet oleg653-client.virtnet PASS 110 (166s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 15:17:30 (1788808650) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 15:17:31 (1788808651) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 15:17:32 (1788808652) lustre-OST0000-osc-ffff942d8817c000 active. lustre-OST0001-osc-ffff942d8817c000 active. lustre-OST0000-osc-ffff942d8817c000 active. lustre-OST0001-osc-ffff942d8817c000 active. MGC192.168.206.153@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 15:17:34 (1788808654) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 15:17:36 (1788808656) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 15:17:37 (1788808657) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 121: trunc append race =================== 15:17:38 (1788808658) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00338319 s, 6.1 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0120882 s, 1.7 MB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (4s) == sanityn test 122: directory size is consistent across mounts ========================================================== 15:17:42 (1788808662) PASS 122 (3s) == sanityn test 200: service remains healthy while able to process request ========================================================== 15:17:45 (1788808665) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.087 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0917 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0889 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0942 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.092 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0876 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0928 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1494 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1571 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1558 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1737 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1672 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1768 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1592 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2532 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2553 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2382 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2519 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff942d8817c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff942d8817c000.ost_server_uuid in FULL state after 0 sec PASS 200 (67s) cleanup: ====================================================== == sanityn test complete, duration 7529 sec ============== 15:18:53 (1788808733) === sanityn: start cleanup 15:18:53 (1788808733) === Stopping clients: oleg653-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg653-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 15:19:40 (1788808780) ===