-----============= acceptance-small: sanityn ============----- Tue Sep 8 08:09:59 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg359-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 08:10:20 (1788869420) === Starting client oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre2 Mount client oleg359-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre2 Started clients oleg359-client.virtnet: 192.168.203.159@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg359-client.virtnet: executing check_config_client /mnt/lustre oleg359-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg359-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e7d12746800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8e7d12dd3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e7d12746800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e7d12dd3800.idle_timeout=debug disable quota as required oleg359-server: oleg359-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 08:10:58 (1788869458) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443838 s, 23.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7604] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 08:11:01 (1788869461) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 08:11:11 (1788869471) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 08:11:19 (1788869479) -rw-r--r-- 1 root root 0 Sep 8 08:11 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 08:11:27 (1788869487) -rw-r--r-- 1 root root 0 Sep 8 08:11 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 08:11:35 (1788869495) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 08:11:42 (1788869502) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 08:11:49 (1788869509) total 1 -rwxrwxrwx 1 root root 0 Sep 8 08:11 f2a -rwxrwxrwx 1 root root 0 Sep 8 08:11 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 8 08:11 f2a -rwxrwxrwx 1 root root 0 Sep 8 08:11 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00975574 s, 52.5 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 08:11:57 (1788869517) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 08:12:01 (1788869521) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.499687 s, 4.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 08:12:09 (1788869529) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 08:12:17 (1788869537) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 08:12:25 (1788869545) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 08:12:33 (1788869553) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 08:12:41 (1788869561) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (8s) == sanityn test 8: remove of open special file on other node ==================================================================== 08:12:49 (1788869569) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 08:12:57 (1788869577) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 08:13:05 (1788869585) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443477 s, 23.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0629119 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 08:13:12 (1788869592) 1+0 records in 1+0 records out 1 byte copied, 0.015097 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0251316 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00710002 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00667096 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0108455 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00784966 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00729968 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00833108 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00855748 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00985636 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00889627 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00808205 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 08:13:20 (1788869600) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0139671 s, 220 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0918609 s, 44.6 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000644755 s, 4.8 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 08:13:27 (1788869607) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 08:13:35 (1788869615) warning: '-runtime' deprecated, use '-t runtime' instead - link 331 (time 1788869628.38 total 10.02 last 33.04) - link 612 (time 1788869638.40 total 20.03 last 28.06) - link 943 (time 1788869648.42 total 30.06 last 33.00) - link 1220 (time 1788869658.43 total 40.07 last 27.68) - link 1523 (time 1788869668.44 total 50.07 last 30.28) - link 1823 (time 1788869678.44 total 60.08 last 29.99) - link 2123 (time 1788869688.44 total 70.08 last 30.00) - link 2421 (time 1788869698.46 total 80.09 last 29.74) - link 2730 (time 1788869708.49 total 90.13 last 30.79) total: 3007 link in 100.00 seconds: 30.07 ops/second using seed 3394892522 running for 100 seconds - stat 4413 (time 1788869630 ; total 11 ; last 11) - stat 9310 (time 1788869641 ; total 22 ; last 11) - stat 10000 (time 1788869642 ; total 23 ; last 1) - stat 15284 (time 1788869653 ; total 34 ; last 11) - stat 19644 (time 1788869664 ; total 45 ; last 11) - stat 20000 (time 1788869664 ; total 45 ; last 0) - stat 23385 (time 1788869675 ; total 56 ; last 11) - stat 25923 (time 1788869686 ; total 67 ; last 11) - stat 28543 (time 1788869697 ; total 78 ; last 11) - stat 30000 (time 1788869702 ; total 83 ; last 5) - stat 32307 (time 1788869713 ; total 94 ; last 11) total: 33734 stats in 100 seconds: 337.339996 stats/second - unlinked 0 (time 1788869669 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3007) error: No such file or directory total: 3006 unlinks in 84 seconds: 35.785713 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21415) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21417) - No such process PASS 12 (144s) == sanityn test 13: test directory page revocation ======= 08:15:59 (1788869759) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 08:16:09 (1788869769) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 22602 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (11s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 08:16:20 (1788869780) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 23182 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (9s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 08:16:29 (1788869789) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23760 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 08:16:37 (1788869797) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24343 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 08:16:45 (1788869805) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24926 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 08:16:55 (1788869815) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (54s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 08:17:50 (1788869870) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.6071 s, 13.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.347516 s, 24.1 MB/s Chance of close/open is 1 in 50 Seed set to 9874 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[0] 1788869875.657246 write 0x1663ad thru 0x1704f3 (0xa147 bytes) 000004[0] 1788869875.690980 write 0x9e2e1a thru 0x9e77ed (0x49d4 bytes) truncating to largest ever: 0x34f3e6 000005[0] 1788869875.715122 trunc from 0x9e77ee to 0x34f3e5 (0x698408 bytes) 000006[0] 1788869875.788743 read 0x158969 thru 0x1669bb (0xe053 bytes) 000007[1] 1788869875.801007 mapwrite 0x56fa29 thru 0x57cf3b (0xd513 bytes) 000010[0] 1788869876.071422 write 0x3ff1ce thru 0x40029b (0x10ce bytes) 000011[1] 1788869876.105769 write 0x04ad60 thru 0x056b28 (0xbdc9 bytes) 000012[1] 1788869876.143517 write 0x592308 thru 0x5a0690 (0xe389 bytes) 000016[1] 1788869876.231827 write 0x361a21 thru 0x362bdb (0x11bb bytes) 000017[0] 1788869876.272605 mapread 0x36ba83 thru 0x3738ec (0x7e6a bytes) 000018[0] 1788869876.393871 read 0x2191e6 thru 0x228122 (0xef3d bytes) 000019[1] 1788869876.422280 mapwrite 0x8fd6fd thru 0x9070cc (0x99d0 bytes) 000020[1] 1788869876.594626 write 0x1ae6fa thru 0x1b9c9d (0xb5a4 bytes) 000021[0] 1788869876.620852 read 0x65a1f2 thru 0x661e4b (0x7c5a bytes) 000022[0] 1788869876.699841 write 0x2f354d thru 0x2f46d6 (0x118a bytes) 000023[1] 1788869876.729946 mapwrite 0x54a21e thru 0x54e256 (0x4039 bytes) 000025[1] 1788869876.955122 mapread 0x04b2eb thru 0x04bde5 (0xafb bytes) 000027[1] 1788869877.018547 mapwrite 0x89dbdf thru 0x8a1118 (0x353a bytes) 000032[1] 1788869877.319663 mapread 0x5f45d1 thru 0x5f9bf7 (0x5627 bytes) 000034[1] 1788869877.369063 mapwrite 0x0cd795 thru 0x0d3901 (0x616d bytes) 000035[1] 1788869877.524319 mapwrite 0x2b7b6b thru 0x2c2b24 (0xafba bytes) 000036[1] 1788869877.770037 write 0x11969c thru 0x11fa90 (0x63f5 bytes) 000037[1] 1788869877.779297 mapread 0x3b55a7 thru 0x3b6279 (0xcd3 bytes) 000038[1] 1788869877.811453 trunc from 0x9070cd to 0x26a7d4 (0x69c8f8 bytes) truncating to largest ever: 0x8747df 000039[0] 1788869877.840756 trunc from 0x26a7d5 to 0x8747de (0x60a00a bytes) 000044[0] 1788869878.003575 mapread 0x39cb07 thru 0x3a0716 (0x3c10 bytes) 000046[1] 1788869878.070950 write 0x28b9c0 thru 0x28d3a3 (0x19e4 bytes) truncating to largest ever: 0x8754e2 000051[0] 1788869878.173167 trunc from 0x8747df to 0x8754e1 (0xd03 bytes) 000054[0] 1788869878.330726 trunc from 0x8754e2 to 0x3d6fb6 (0x49e52b bytes) 000061[0] 1788869878.662968 read 0x3138d3 thru 0x32370b (0xfe39 bytes) 000064[1] 1788869878.887846 mapread 0x19e370 thru 0x1a9eaa (0xbb3b bytes) 000065[1] 1788869879.079433 write 0x968e71 thru 0x96d1fa (0x438a bytes) 000067[0] 1788869879.125659 mapwrite 0x365c9d thru 0x367ae1 (0x1e45 bytes) 000070[1] 1788869879.251373 mapread 0x638bc6 thru 0x63e303 (0x573e bytes) 000071[1] 1788869879.348463 mapwrite 0x4e221e thru 0x4edb06 (0xb8e9 bytes) 000073[1] 1788869879.703567 mapread 0x875586 thru 0x884f88 (0xfa03 bytes) 000078[0] 1788869879.780863 mapread 0x621402 thru 0x624688 (0x3287 bytes) 000083[0] 1788869879.875783 trunc from 0x96d1fb to 0x372fc2 (0x5fa238 bytes) 000085[1] 1788869879.957623 trunc from 0x372fc3 to 0x14f9b7 (0x22360b bytes) 000087[1] 1788869880.046621 trunc from 0x14f9b8 to 0x375d25 (0x22636e bytes) 000092[1] 1788869880.146753 mapread 0x32f543 thru 0x337126 (0x7be4 bytes) 000095[0] 1788869880.225004 mapread 0x1002c7 thru 0x103175 (0x2eaf bytes) 000098[0] 1788869880.293616 mapwrite 0x84d779 thru 0x85817d (0xaa05 bytes) 000099[0] 1788869880.700050 mapwrite 0x2303da thru 0x23a184 (0x9dab bytes) 000100[0] 1788869880.851087 mapread 0x179c47 thru 0x180f21 (0x72db bytes) 000102[1] 1788869880.921108 write 0x40dc49 thru 0x41abe7 (0xcf9f bytes) 000103[0] 1788869880.982560 read 0x851669 thru 0x85562e (0x3fc6 bytes) 000107[0] 1788869881.049418 read 0x3f1a9e thru 0x3fe42d (0xc990 bytes) 000108[0] 1788869881.077811 read 0x31470f thru 0x31476c (0x5e bytes) 000109[0] 1788869881.099638 read 0x7ccb7d thru 0x7cffe7 (0x346b bytes) 000113[0] 1788869881.153051 trunc from 0x85817e to 0x2d8ddd (0x57f3a0 bytes) 000116[0] 1788869881.225362 write 0x42e23a thru 0x43afd0 (0xcd97 bytes) 000121[1] 1788869881.340527 trunc from 0x43afd1 to 0x2d6ffe (0x163fd2 bytes) 000122[1] 1788869881.524507 read 0x1c0833 thru 0x1c579f (0x4f6d bytes) 000123[1] 1788869881.588418 trunc from 0x2d6fff to 0x57a09c (0x2a309e bytes) 000124[1] 1788869881.669959 write 0x568256 thru 0x56aa36 (0x27e1 bytes) 000126[0] 1788869881.752777 write 0x74a1da thru 0x74c4a0 (0x22c7 bytes) 000128[0] 1788869881.811055 read 0x0d3079 thru 0x0d392d (0x8b5 bytes) 000129[0] 1788869881.829676 read 0x44890f thru 0x44e9cb (0x60bd bytes) 000131[0] 1788869881.870521 mapread 0x00c2cb thru 0x0170c5 (0xadfb bytes) 000133[1] 1788869881.926479 mapwrite 0x3a6bf1 thru 0x3a7dac (0x11bc bytes) 000134[0] 1788869882.002570 trunc from 0x74c4a1 to 0x12b37c (0x621124 bytes) 000135[0] 1788869882.072838 mapwrite 0x95b207 thru 0x961084 (0x5e7e bytes) 000140[0] 1788869882.217917 mapwrite 0x4e9e3e thru 0x4f2a8c (0x8c4f bytes) 000145[1] 1788869882.370860 trunc from 0x961085 to 0x6c9104 (0x297f80 bytes) 000150[1] 1788869882.475566 mapread 0x140cc5 thru 0x140d34 (0x70 bytes) 000151[0] 1788869882.480664 read 0x33547e thru 0x33dc04 (0x8787 bytes) 000153[0] 1788869882.528285 read 0x466c8e thru 0x46a2b3 (0x3626 bytes) 000156[1] 1788869882.551556 read 0x4be4fe thru 0x4ca9b9 (0xc4bc bytes) 000157[0] 1788869882.559259 read 0x4d4887 thru 0x4de836 (0x9fb0 bytes) 000158[1] 1788869882.571885 mapwrite 0x499e34 thru 0x4a3595 (0x9762 bytes) 000160[0] 1788869882.708555 mapread 0x34d976 thru 0x35a349 (0xc9d4 bytes) 000161[0] 1788869882.760558 read 0x6755c5 thru 0x678c10 (0x364c bytes) 000162[1] 1788869882.951513 write 0x0271c5 thru 0x02a5a1 (0x33dd bytes) 000165[1] 1788869883.043826 trunc from 0x6c9105 to 0x6d5821 (0xc71d bytes) 000166[1] 1788869883.121671 read 0x2983a5 thru 0x29fed0 (0x7b2c bytes) 000168[0] 1788869883.140284 trunc from 0x6d5822 to 0x457cc7 (0x27db5a bytes) 000169[1] 1788869883.204922 mapwrite 0x32846c thru 0x3292a0 (0xe35 bytes) 000173[0] 1788869883.427042 mapread 0x3e51c2 thru 0x3f4291 (0xf0d0 bytes) 000175[1] 1788869883.686043 trunc from 0x457cc8 to 0x7d8e5f (0x381198 bytes) truncating to largest ever: 0x9b076c 000176[0] 1788869883.756594 trunc from 0x7d8e60 to 0x9b076b (0x1d790c bytes) 000177[0] 1788869883.874400 write 0x999e3e thru 0x99ce86 (0x3049 bytes) 000180[1] 1788869884.052364 trunc from 0x9b076c to 0x547c4b (0x468b20 bytes) 000182[1] 1788869884.216772 mapwrite 0x3f1a1f thru 0x3fe67e (0xcc60 bytes) 000187[0] 1788869884.337896 mapwrite 0x89f248 thru 0x8a07d9 (0x1592 bytes) 000191[0] 1788869884.467934 read 0x1a5fa9 thru 0x1ac6ed (0x6745 bytes) 000193[0] 1788869884.477224 read 0x28c372 thru 0x28f44d (0x30dc bytes) 000194[1] 1788869884.504231 read 0x56d99f thru 0x57c771 (0xedd3 bytes) 000196[0] 1788869884.563533 mapwrite 0x2214ae thru 0x225b93 (0x46e6 bytes) 000197[1] 1788869884.664113 read 0x43f137 thru 0x4405b6 (0x1480 bytes) 000199[1] 1788869884.682991 trunc from 0x8a07da to 0x4c23cc (0x3de40d bytes) 000200[0] 1788869884.747509 mapread 0x0eed3b thru 0x0f7951 (0x8c17 bytes) 000201[0] 1788869884.813242 mapread 0x200c9b thru 0x206f2e (0x6294 bytes) 000202[0] 1788869884.856668 mapread 0x05def6 thru 0x065c86 (0x7d91 bytes) 000204[1] 1788869884.901392 mapread 0x381da0 thru 0x389350 (0x75b1 bytes) 000210[0] 1788869884.946705 mapwrite 0x701f8a thru 0x70fcea (0xdd61 bytes) 000214[0] 1788869885.217449 read 0x52c4e0 thru 0x53638b (0x9eac bytes) truncating to largest ever: 0x9b8451 000218[1] 1788869885.281957 trunc from 0x70fceb to 0x9b8450 (0x2a8766 bytes) 000219[1] 1788869885.327601 mapread 0x7d66a6 thru 0x7d7ecd (0x1828 bytes) 000220[1] 1788869885.356818 mapwrite 0x2d8041 thru 0x2e7590 (0xf550 bytes) 000221[0] 1788869885.457014 write 0x93c271 thru 0x9408b0 (0x4640 bytes) 000222[1] 1788869885.487672 trunc from 0x9b8451 to 0x276046 (0x74240a bytes) 000224[0] 1788869885.525019 read 0x189a84 thru 0x18a83e (0xdbb bytes) 000225[1] 1788869885.557817 write 0x84793d thru 0x850721 (0x8de5 bytes) 000227[0] 1788869885.580436 mapread 0x3585be thru 0x35a1ad (0x1bf0 bytes) 000230[1] 1788869885.685599 trunc from 0x850722 to 0x11e91b (0x731e06 bytes) 000231[0] 1788869885.774509 write 0x394900 thru 0x39ab3c (0x623d bytes) 000232[1] 1788869885.812783 mapwrite 0x27ccc8 thru 0x280e8f (0x41c8 bytes) 000233[0] 1788869885.888923 read 0x1e30dd thru 0x1ea223 (0x7147 bytes) 000235[0] 1788869885.985180 trunc from 0x39ab3d to 0x5dadf1 (0x2402b5 bytes) 000238[0] 1788869886.126635 mapwrite 0x015615 thru 0x016c04 (0x15f0 bytes) 000240[1] 1788869886.299879 trunc from 0x5dadf2 to 0x7948ff (0x1b9b0e bytes) 000241[0] 1788869886.349504 mapread 0x68e589 thru 0x6960f6 (0x7b6e bytes) 000242[1] 1788869886.521113 trunc from 0x794900 to 0x628596 (0x16c369 bytes) 000243[0] 1788869886.636369 read 0x35bb23 thru 0x367118 (0xb5f6 bytes) 000246[1] 1788869886.692385 write 0x7888d7 thru 0x79166e (0x8d98 bytes) 000247[1] 1788869886.698878 mapwrite 0x3baf05 thru 0x3c796d (0xca69 bytes) 000248[0] 1788869886.887648 write 0x359fef thru 0x35fd7a (0x5d8c bytes) 000251[0] 1788869886.992560 mapread 0x149ba0 thru 0x14b281 (0x16e2 bytes) 000252[1] 1788869887.078661 mapread 0x758a0c thru 0x7619c7 (0x8fbc bytes) 000253[0] 1788869887.124088 read 0x77cae3 thru 0x7815c3 (0x4ae1 bytes) 000254[0] 1788869887.174227 read 0x00386e thru 0x003f78 (0x70b bytes) 000256[1] 1788869887.182553 write 0x615034 thru 0x615b3a (0xb07 bytes) 000257[0] 1788869887.216959 trunc from 0x79166f to 0x315575 (0x47c0f9 bytes) 000258[0] 1788869887.307605 mapread 0x1b7f7f thru 0x1c112a (0x91ac bytes) 000259[0] 1788869887.362498 trunc from 0x315576 to 0x33435d (0x1ede8 bytes) 000260[1] 1788869887.410714 read 0x050f5c thru 0x0591fb (0x82a0 bytes) 000264[1] 1788869887.538301 write 0x8c4a41 thru 0x8c7fb7 (0x3577 bytes) 000265[1] 1788869887.582632 trunc from 0x8c7fb8 to 0x240c4a (0x68736d bytes) 000266[1] 1788869887.615223 mapwrite 0x310bfc thru 0x3163ef (0x57f4 bytes) 000267[0] 1788869887.847474 read 0x1d6f3c thru 0x1da079 (0x313e bytes) 000273[0] 1788869888.127069 mapwrite 0x4ee116 thru 0x4f89e8 (0xa8d3 bytes) 000276[0] 1788869888.477477 read 0x2ce0e8 thru 0x2d495b (0x6874 bytes) 000279[0] 1788869888.564663 write 0x464c3c thru 0x471359 (0xc71e bytes) 000285[0] 1788869888.677428 mapwrite 0x56276b thru 0x5628e9 (0x17f bytes) 000287[1] 1788869888.838068 mapwrite 0x0e8e73 thru 0x0ef07c (0x620a bytes) 000293[0] 1788869889.244055 read 0x0c9d40 thru 0x0d8c34 (0xeef5 bytes) 000294[1] 1788869889.266449 read 0x1b82c6 thru 0x1beafc (0x6837 bytes) 000295[1] 1788869889.280230 write 0x12483e thru 0x12c43a (0x7bfd bytes) 000298[1] 1788869889.360994 read 0x342ac2 thru 0x343207 (0x746 bytes) 000299[0] 1788869889.386929 mapread 0x2bb1e1 thru 0x2c4009 (0x8e29 bytes) 000301[1] 1788869889.460323 mapread 0x454fe3 thru 0x4634f5 (0xe513 bytes) 000306[0] 1788869889.558518 mapread 0x37d4a6 thru 0x386f73 (0x9ace bytes) 000307[0] 1788869889.607488 trunc from 0x5628ea to 0x609f73 (0xa768a bytes) 000308[0] 1788869889.663971 read 0x535602 thru 0x54407d (0xea7c bytes) 000310[1] 1788869889.702834 write 0x8bb762 thru 0x8c2240 (0x6adf bytes) 000311[0] 1788869889.728969 write 0x7c1280 thru 0x7c4d46 (0x3ac7 bytes) 000313[1] 1788869889.759959 read 0x1105a3 thru 0x11bdd7 (0xb835 bytes) 000314[0] 1788869889.783015 trunc from 0x8c2241 to 0x2b0aa1 (0x61179f bytes) 000315[1] 1788869889.855689 trunc from 0x2b0aa2 to 0x4861ad (0x1d570c bytes) 000316[0] 1788869889.894634 mapwrite 0x03f3ce thru 0x04c888 (0xd4bb bytes) 000317[1] 1788869890.054648 write 0x265b0a thru 0x27473b (0xec32 bytes) 000318[0] 1788869890.143048 trunc from 0x4861ae to 0x865b27 (0x3df97a bytes) 000319[0] 1788869890.234478 mapread 0x025c98 thru 0x02f13a (0x94a3 bytes) 000320[0] 1788869890.253733 mapread 0x462e4c thru 0x4692cc (0x6481 bytes) 000321[0] 1788869890.300006 mapwrite 0x89da78 thru 0x8ad041 (0xf5ca bytes) 000323[0] 1788869890.461642 trunc from 0x8ad042 to 0x063da7 (0x84929a bytes) 000333[0] 1788869890.755032 mapread 0x0149c4 thru 0x0198a3 (0x4ee0 bytes) 000334[1] 1788869890.782975 write 0x594b65 thru 0x597337 (0x27d3 bytes) 000335[0] 1788869890.822319 write 0x6ac64e thru 0x6b3912 (0x72c5 bytes) 000336[0] 1788869890.862757 read 0x3f16a1 thru 0x3fed60 (0xd6c0 bytes) 000337[0] 1788869890.936529 trunc from 0x6b3913 to 0x0db2a7 (0x5d866b bytes) 000338[1] 1788869890.960502 mapwrite 0x139df7 thru 0x13e078 (0x4282 bytes) 000341[1] 1788869891.227277 trunc from 0x13e079 to 0x56ed70 (0x430cf8 bytes) 000344[0] 1788869891.370295 mapwrite 0x332884 thru 0x33290c (0x89 bytes) 000345[0] 1788869891.539546 mapread 0x35796a thru 0x35ba8d (0x4124 bytes) 000346[1] 1788869891.582782 mapread 0x0f2b43 thru 0x0f9f7a (0x7438 bytes) 000348[1] 1788869891.703498 mapwrite 0x1c838c thru 0x1c9c9f (0x1914 bytes) 000349[1] 1788869891.856018 trunc from 0x56ed71 to 0x51f187 (0x4fbe9 bytes) 000350[0] 1788869891.983726 read 0x36c9c8 thru 0x37b3c5 (0xe9fe bytes) 000352[1] 1788869892.129817 trunc from 0x51f188 to 0x099c78 (0x48550f bytes) 000353[1] 1788869892.219261 trunc from 0x099c79 to 0x2f43b5 (0x25a73d bytes) 000354[0] 1788869892.252513 mapread 0x29fb6f thru 0x2a3d74 (0x4206 bytes) 000357[1] 1788869892.351360 mapread 0x186140 thru 0x1945e5 (0xe4a6 bytes) 000358[1] 1788869892.404362 trunc from 0x2f43b6 to 0x1973fa (0x15cfbb bytes) 000361[0] 1788869892.693641 write 0x0783ea thru 0x0793a4 (0xfbb bytes) 000363[1] 1788869892.783065 read 0x000655 thru 0x00e194 (0xdb40 bytes) 000364[1] 1788869892.860417 read 0x021687 thru 0x0312bc (0xfc36 bytes) 000365[0] 1788869892.874787 trunc from 0x1973fb to 0x0f1fd9 (0xa5421 bytes) 000366[1] 1788869892.990218 mapwrite 0x78819b thru 0x78f8ad (0x7713 bytes) 000367[0] 1788869893.366351 trunc from 0x78f8ae to 0x6e313d (0xac770 bytes) 000368[0] 1788869893.428356 read 0x691bf1 thru 0x69383f (0x1c4f bytes) 000369[1] 1788869893.477650 mapwrite 0x023e25 thru 0x029601 (0x57dd bytes) 000370[1] 1788869893.633322 mapwrite 0x20e29a thru 0x219197 (0xaefe bytes) 000375[1] 1788869893.881745 mapwrite 0x27b9c6 thru 0x281e8b (0x64c6 bytes) 000377[0] 1788869894.037977 mapwrite 0x256d57 thru 0x25d889 (0x6b33 bytes) 000378[0] 1788869894.197922 write 0x26f435 thru 0x277634 (0x8200 bytes) 000380[1] 1788869894.202726 write 0x419e83 thru 0x428064 (0xe1e2 bytes) 000381[1] 1788869894.250753 write 0x46192f thru 0x46a7f1 (0x8ec3 bytes) 000382[1] 1788869894.306025 mapwrite 0x1df38b thru 0x1eae07 (0xba7d bytes) 000387[0] 1788869894.646346 write 0x37d757 thru 0x380c8e (0x3538 bytes) 000388[1] 1788869894.686696 mapread 0x424b22 thru 0x42b1b7 (0x6696 bytes) 000391[0] 1788869894.786464 mapread 0x0c0996 thru 0x0ca0e2 (0x974d bytes) 000392[1] 1788869894.841766 write 0x05f65b thru 0x06850f (0x8eb5 bytes) 000399[1] 1788869895.016126 mapwrite 0x87b71c thru 0x87bb49 (0x42e bytes) 000400[0] 1788869895.218601 read 0x1875b6 thru 0x18fb38 (0x8583 bytes) 000401[1] 1788869895.265891 read 0x7e31f1 thru 0x7e86be (0x54ce bytes) 000403[0] 1788869895.278118 mapwrite 0x8789d8 thru 0x87e2ed (0x5916 bytes) 000404[1] 1788869895.578566 trunc from 0x87e2ee to 0x8a9783 (0x2b496 bytes) 000406[0] 1788869895.713291 mapread 0x0150e9 thru 0x01963e (0x4556 bytes) 000409[1] 1788869895.818886 mapwrite 0x4053a3 thru 0x40ae5f (0x5abd bytes) 000410[1] 1788869896.023654 read 0x795944 thru 0x7a35d6 (0xdc93 bytes) 000413[1] 1788869896.091397 mapread 0x5f15e7 thru 0x5fc0e2 (0xaafc bytes) 000415[1] 1788869896.157458 mapread 0x4c975f thru 0x4c979f (0x41 bytes) 000416[0] 1788869896.184253 write 0x7736ce thru 0x77d8f6 (0xa229 bytes) 000418[0] 1788869896.239480 mapread 0x769f34 thru 0x76b5f8 (0x16c5 bytes) 000419[0] 1788869896.264984 read 0x66e4cc thru 0x67b866 (0xd39b bytes) 000424[0] 1788869896.324668 mapread 0x2e5d44 thru 0x2f5806 (0xfac3 bytes) 000426[0] 1788869896.417604 write 0x43a492 thru 0x4464a7 (0xc016 bytes) 000427[1] 1788869896.434520 mapread 0x304eb3 thru 0x309060 (0x41ae bytes) 000428[0] 1788869896.489505 read 0x788c3d thru 0x7953f4 (0xc7b8 bytes) 000430[0] 1788869896.523230 trunc from 0x8a9784 to 0x7b2d11 (0xf6a72 bytes) 000436[1] 1788869896.630965 read 0x0dd637 thru 0x0e1038 (0x3a02 bytes) 000440[1] 1788869896.688239 mapwrite 0x972513 thru 0x979d3d (0x782b bytes) 000441[0] 1788869896.877437 mapwrite 0x4f5c34 thru 0x4f698a (0xd57 bytes) 000442[0] 1788869897.013259 trunc from 0x979d3e to 0x4549b7 (0x525386 bytes) 000443[1] 1788869897.047464 mapread 0x0e2793 thru 0x0f12d1 (0xeb3f bytes) 000444[1] 1788869897.124951 write 0x6bed9b thru 0x6c8f61 (0xa1c7 bytes) 000445[1] 1788869897.158256 write 0x4732bd thru 0x47afc2 (0x7d06 bytes) 000446[1] 1788869897.170780 mapwrite 0x26e3f9 thru 0x277ebe (0x9ac6 bytes) 000447[1] 1788869897.323899 write 0x916f92 thru 0x9202a1 (0x9310 bytes) 000451[1] 1788869897.387946 read 0x3b6349 thru 0x3b67bc (0x474 bytes) 000455[1] 1788869897.511611 mapwrite 0x55e14d thru 0x565acc (0x7980 bytes) 000456[0] 1788869897.637316 write 0x52829d thru 0x535f11 (0xdc75 bytes) 000459[0] 1788869897.710411 read 0x548347 thru 0x55215e (0x9e18 bytes) 000460[1] 1788869897.751333 mapread 0x3ccd07 thru 0x3d0e22 (0x411c bytes) 000463[0] 1788869897.853972 mapread 0x897b32 thru 0x89ee3b (0x730a bytes) 000464[1] 1788869897.901746 read 0x9045c5 thru 0x906c1b (0x2657 bytes) 000466[1] 1788869897.922920 write 0x81dc9d thru 0x828704 (0xaa68 bytes) 000468[1] 1788869897.977440 mapwrite 0x61fa51 thru 0x62f137 (0xf6e7 bytes) 000469[1] 1788869898.233390 read 0x52b08f thru 0x53355c (0x84ce bytes) 000470[1] 1788869898.268788 trunc from 0x9202a2 to 0x1d2190 (0x74e111 bytes) 000471[1] 1788869898.298919 mapread 0x0b823f thru 0x0ba202 (0x1fc4 bytes) 000472[1] 1788869898.348951 mapread 0x193c6c thru 0x1948c4 (0xc59 bytes) 000473[1] 1788869898.359122 read 0x1250e1 thru 0x130e6f (0xbd8f bytes) 000478[1] 1788869898.564146 mapwrite 0x63b61d thru 0x640b76 (0x555a bytes) 000479[1] 1788869898.927863 read 0x50473a thru 0x514140 (0xfa07 bytes) 000480[1] 1788869898.965335 mapread 0x073b29 thru 0x07f2c5 (0xb79d bytes) 000481[0] 1788869898.995946 mapread 0x314a45 thru 0x31aa2b (0x5fe7 bytes) 000483[0] 1788869899.057399 mapread 0x34420e thru 0x34ecf1 (0xaae4 bytes) 000486[1] 1788869899.139561 write 0x0fc409 thru 0x10805b (0xbc53 bytes) 000487[1] 1788869899.224674 mapread 0x2ecaa9 thru 0x2fa382 (0xd8da bytes) 000488[1] 1788869899.296671 trunc from 0x640b77 to 0x8b0145 (0x26f5cf bytes) 000489[0] 1788869899.380616 mapwrite 0x495138 thru 0x4a190b (0xc7d4 bytes) 000491[1] 1788869899.558585 write 0x36c5b4 thru 0x36e8ca (0x2317 bytes) 000492[0] 1788869899.581762 mapwrite 0x594bbd thru 0x597c20 (0x3064 bytes) 000496[1] 1788869899.743055 write 0x55424d thru 0x55b1ad (0x6f61 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 9899 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000007[0] 1788869900.719645 mapwrite 0x423000 thru 0x4319f3 (0xe9f4 bytes) 000008[0] 1788869900.946412 read 0x38f000 thru 0x39665d (0x765e bytes) 000009[1] 1788869900.980056 mapread 0x1b1000 thru 0x1b5d34 (0x4d35 bytes) 000010[1] 1788869901.108611 read_OD 0x0fe000 thru 0x104fff (0x7000 bytes) 000011[1] 1788869901.144308 mapwrite 0x979000 thru 0x982e78 (0x9e79 bytes) 000012[1] 1788869901.356603 mapread 0x571000 thru 0x57bbb5 (0xabb6 bytes) 000013[1] 1788869901.399099 write_OD 0x3d5000 thru 0x3d7fff (0x3000 bytes) 000014[0] 1788869901.471371 write 0x59b000 thru 0x5a3155 (0x8156 bytes) 000017[1] 1788869901.505875 mapwrite 0x49d000 thru 0x4a9cc7 (0xccc8 bytes) 000018[0] 1788869901.609526 write 0x68f000 thru 0x699b26 (0xab27 bytes) 000021[1] 1788869901.676904 write_OD 0x686000 thru 0x691fff (0xc000 bytes) 000023[1] 1788869901.783419 read_OD 0x14c000 thru 0x15afff (0xf000 bytes) truncating to largest ever: 0x43e206 000024[1] 1788869901.796691 trunc from 0x982e79 to 0x43e205 (0x544c73 bytes) 000028[1] 1788869901.845602 write_OD 0x77a000 thru 0x780fff (0x7000 bytes) 000030[0] 1788869901.903023 mapread 0x6fa000 thru 0x6fde40 (0x3e41 bytes) 000031[0] 1788869901.929073 mapwrite 0x273000 thru 0x27ca76 (0x9a77 bytes) 000036[0] 1788869902.119025 read 0x62b000 thru 0x62d61f (0x2620 bytes) 000038[0] 1788869902.168913 mapwrite 0x0dd000 thru 0x0ebc9d (0xec9e bytes) 000042[1] 1788869902.404272 write_OD 0x341000 thru 0x34ffff (0xf000 bytes) truncating to largest ever: 0x689fc3 000043[1] 1788869902.522058 trunc from 0x781000 to 0x689fc2 (0xf703d bytes) 000044[0] 1788869902.613554 read 0x07f000 thru 0x0822ef (0x32f0 bytes) 000045[0] 1788869902.648760 mapwrite 0x850000 thru 0x859fd6 (0x9fd7 bytes) 000046[1] 1788869902.856781 read_OD 0x662000 thru 0x665fff (0x4000 bytes) 000047[0] 1788869902.929844 read 0x711000 thru 0x720be3 (0xfbe4 bytes) 000048[0] 1788869902.950755 read 0x3ce000 thru 0x3d4c13 (0x6c14 bytes) 000050[0] 1788869902.968694 mapwrite 0x7c1000 thru 0x7c35b4 (0x25b5 bytes) 000052[0] 1788869903.165370 mapwrite 0x720000 thru 0x72fbf1 (0xfbf2 bytes) 000053[1] 1788869903.347122 read_OD 0x605000 thru 0x60dfff (0x9000 bytes) 000054[1] 1788869903.386476 read_OD 0x38b000 thru 0x38cfff (0x2000 bytes) 000055[1] 1788869903.426634 write_OD 0x40b000 thru 0x418fff (0xe000 bytes) truncating to largest ever: 0x710d7b 000063[1] 1788869903.511592 trunc from 0x859fd7 to 0x710d7a (0x14925c bytes) 000065[0] 1788869903.587620 mapwrite 0x42e000 thru 0x435114 (0x7115 bytes) 000066[0] 1788869903.817002 mapwrite 0x2f3000 thru 0x301bb7 (0xebb8 bytes) 000067[1] 1788869903.994011 mapwrite 0x07d000 thru 0x082f58 (0x5f59 bytes) 000068[1] 1788869904.227530 read_OD 0x52d000 thru 0x539fff (0xd000 bytes) 000069[1] 1788869904.288871 read_OD 0x400000 thru 0x401fff (0x2000 bytes) 000071[1] 1788869904.310924 mapread 0x609000 thru 0x609da3 (0xda4 bytes) 000072[1] 1788869904.325941 read_OD 0x6da000 thru 0x6e5fff (0xc000 bytes) 000073[0] 1788869904.335636 mapwrite 0x059000 thru 0x06139f (0x83a0 bytes) 000074[1] 1788869904.422510 mapwrite 0x8dd000 thru 0x8e321b (0x621c bytes) truncating to largest ever: 0x982c17 000075[1] 1788869904.534437 trunc from 0x8e321c to 0x982c16 (0x9f9fb bytes) 000080[0] 1788869904.549114 write 0x714000 thru 0x714a11 (0xa12 bytes) 000081[0] 1788869904.581726 write 0x5bb000 thru 0x5c94a1 (0xe4a2 bytes) 000083[0] 1788869904.596630 mapread 0x439000 thru 0x4474cb (0xe4cc bytes) 000087[0] 1788869904.663469 write 0x14f000 thru 0x1510c5 (0x20c6 bytes) 000090[1] 1788869904.709057 read_OD 0x34f000 thru 0x356fff (0x8000 bytes) 000091[0] 1788869904.758368 write 0x216000 thru 0x223fe8 (0xdfe9 bytes) 000092[0] 1788869904.779174 read 0x3e7000 thru 0x3eddb4 (0x6db5 bytes) 000094[0] 1788869904.810595 trunc from 0x982c17 to 0x5372f1 (0x44b925 bytes) 000097[1] 1788869904.879872 trunc from 0x5372f2 to 0x42064f (0x116ca2 bytes) 000098[0] 1788869904.968896 mapwrite 0x428000 thru 0x4312b3 (0x92b4 bytes) 000099[0] 1788869905.118033 trunc from 0x4312b4 to 0x32098b (0x110928 bytes) 000100[0] 1788869905.144135 trunc from 0x32098c to 0x5c612c (0x2a57a1 bytes) 000101[0] 1788869905.187104 write 0x839000 thru 0x83bc0a (0x2c0b bytes) 000102[0] 1788869905.198023 mapwrite 0x373000 thru 0x3796de (0x66df bytes) 000103[1] 1788869905.365719 trunc from 0x83bc0b to 0x7ed56c (0x4e69e bytes) 000105[1] 1788869905.458687 mapread 0x740000 thru 0x74aa96 (0xaa97 bytes) 000109[0] 1788869905.511827 mapread 0x3aa000 thru 0x3aafb6 (0xfb7 bytes) 000111[0] 1788869905.553580 trunc from 0x7ed56d to 0x873ea9 (0x8693d bytes) 000114[1] 1788869905.647833 mapwrite 0x810000 thru 0x812a2b (0x2a2c bytes) 000115[1] 1788869905.762262 mapwrite 0x74d000 thru 0x74d651 (0x652 bytes) 000116[0] 1788869905.851396 trunc from 0x873eaa to 0x59a5e9 (0x2d98c0 bytes) 000118[0] 1788869905.891173 trunc from 0x59a5ea to 0x48fcf0 (0x10a8f9 bytes) 000119[0] 1788869905.908892 mapread 0x2fa000 thru 0x30993f (0xf940 bytes) 000121[1] 1788869905.959921 mapwrite 0x78b000 thru 0x7911bf (0x61c0 bytes) 000124[1] 1788869906.128567 mapread 0x53c000 thru 0x53c406 (0x407 bytes) 000126[1] 1788869906.156286 mapread 0x37a000 thru 0x388b1e (0xeb1f bytes) 000129[0] 1788869906.205185 mapwrite 0x547000 thru 0x551a16 (0xaa17 bytes) 000131[0] 1788869906.324828 read 0x330000 thru 0x338ece (0x8ecf bytes) 000132[1] 1788869906.337743 mapwrite 0x8e8000 thru 0x8eff99 (0x7f9a bytes) 000133[1] 1788869906.524085 mapread 0x36a000 thru 0x373a91 (0x9a92 bytes) 000134[0] 1788869906.558147 read 0x884000 thru 0x890669 (0xc66a bytes) 000135[0] 1788869906.579710 read 0x6a0000 thru 0x6a0edb (0xedc bytes) 000138[0] 1788869906.584718 write 0x3bb000 thru 0x3bfb8b (0x4b8c bytes) 000142[0] 1788869906.616659 write 0x1f3000 thru 0x1ffcf0 (0xccf1 bytes) 000143[1] 1788869906.662483 trunc from 0x8eff9a to 0x4e85a3 (0x4079f6 bytes) 000144[0] 1788869906.716171 mapwrite 0x221000 thru 0x224bea (0x3beb bytes) 000145[0] 1788869906.841793 mapread 0x4b7000 thru 0x4c4ed4 (0xded5 bytes) 000149[1] 1788869906.914999 trunc from 0x4e85a4 to 0x5d16f7 (0xe9154 bytes) 000150[0] 1788869906.955075 trunc from 0x5d16f8 to 0x35bfa4 (0x275753 bytes) 000153[0] 1788869907.024676 mapread 0x1f8000 thru 0x204123 (0xc124 bytes) 000154[0] 1788869907.072081 trunc from 0x35bfa5 to 0x303b7a (0x5842a bytes) 000156[1] 1788869907.095345 read_OD 0x211000 thru 0x215fff (0x5000 bytes) 000157[1] 1788869907.129758 mapwrite 0x21e000 thru 0x22a5d3 (0xc5d4 bytes) 000158[0] 1788869907.279636 trunc from 0x303b7b to 0x8c62e7 (0x5c276d bytes) 000159[1] 1788869907.353471 write_OD 0x83e000 thru 0x849fff (0xc000 bytes) 000164[1] 1788869907.511585 write_OD 0x57a000 thru 0x57afff (0x1000 bytes) 000166[0] 1788869907.606593 read 0x42e000 thru 0x431a48 (0x3a49 bytes) 000169[1] 1788869907.615636 mapwrite 0x16d000 thru 0x17a2d9 (0xd2da bytes) 000170[1] 1788869907.739922 mapread 0x662000 thru 0x664047 (0x2048 bytes) 000171[0] 1788869907.770609 trunc from 0x8c62e8 to 0x96cf91 (0xa6caa bytes) 000172[1] 1788869907.807362 mapwrite 0x18a000 thru 0x199b7d (0xfb7e bytes) 000173[0] 1788869908.091675 write 0x984000 thru 0x989ade (0x5adf bytes) 000174[0] 1788869908.138952 trunc from 0x989adf to 0x1c6069 (0x7c3a75 bytes) 000176[0] 1788869908.183638 read 0x07b000 thru 0x07eb53 (0x3b54 bytes) 000179[1] 1788869908.222642 mapread 0x06c000 thru 0x06f5da (0x35db bytes) 000181[0] 1788869908.256796 read 0x0ed000 thru 0x0fa9a0 (0xd9a1 bytes) 000182[1] 1788869908.315542 mapread 0x00e000 thru 0x01999d (0xb99e bytes) 000184[0] 1788869908.338328 read 0x068000 thru 0x06ecd0 (0x6cd1 bytes) 000186[0] 1788869908.339579 read 0x025000 thru 0x02aaaa (0x5aab bytes) 000187[1] 1788869908.340333 write_OD 0x699000 thru 0x6a5fff (0xd000 bytes) 000189[0] 1788869908.488507 write 0x75c000 thru 0x76975e (0xd75f bytes) 000192[0] 1788869908.556608 trunc from 0x76975f to 0x4f3567 (0x2761f7 bytes) 000193[1] 1788869908.589610 trunc from 0x4f3568 to 0x66321a (0x16fcb3 bytes) 000200[1] 1788869908.689755 mapread 0x3c2000 thru 0x3d0298 (0xe299 bytes) 000201[1] 1788869908.722323 trunc from 0x66321b to 0x94145f (0x2de245 bytes) 000202[0] 1788869908.774411 write 0x8c7000 thru 0x8cd9e9 (0x69ea bytes) 000203[0] 1788869908.874794 trunc from 0x941460 to 0x5f6a8c (0x34a9d3 bytes) 000205[1] 1788869908.910831 mapread 0x0b1000 thru 0x0b29d8 (0x19d9 bytes) 000206[0] 1788869908.942086 mapwrite 0x932000 thru 0x93bdc9 (0x9dca bytes) 000209[1] 1788869909.162583 mapwrite 0x527000 thru 0x535388 (0xe389 bytes) 000210[0] 1788869909.268065 trunc from 0x93bdca to 0x61a290 (0x321b39 bytes) 000211[1] 1788869909.301245 write_OD 0x84e000 thru 0x85bfff (0xe000 bytes) 000212[1] 1788869909.363184 mapread 0x5a2000 thru 0x5ae191 (0xc192 bytes) 000215[0] 1788869909.385473 trunc from 0x85c000 to 0x7cc2c1 (0x8fd3e bytes) 000216[1] 1788869909.447134 mapwrite 0x6ab000 thru 0x6bac1b (0xfc1c bytes) 000222[0] 1788869909.747688 write 0x463000 thru 0x4729fb (0xf9fc bytes) 000223[1] 1788869909.779280 mapread 0x106000 thru 0x10c71a (0x671b bytes) 000224[1] 1788869909.848629 write_OD 0x24e000 thru 0x255fff (0x8000 bytes) 000225[1] 1788869909.926628 trunc from 0x7cc2c2 to 0x3011be (0x4cb103 bytes) 000226[1] 1788869909.974465 read_OD 0x268000 thru 0x26afff (0x3000 bytes) 000229[0] 1788869910.001655 mapread 0x2e8000 thru 0x2e8234 (0x235 bytes) 000231[0] 1788869910.008070 trunc from 0x3011bf to 0x1c2bfb (0x13e5c3 bytes) 000232[1] 1788869910.049082 write_OD 0x993000 thru 0x9a0fff (0xe000 bytes) 000233[0] 1788869910.106358 mapwrite 0x890000 thru 0x893adb (0x3adc bytes) 000236[0] 1788869910.197365 trunc from 0x9a1000 to 0x6d58fe (0x2cb701 bytes) 000238[0] 1788869910.217123 trunc from 0x6d58ff to 0x27f8de (0x456020 bytes) truncating to largest ever: 0x9e67c1 000240[0] 1788869910.257271 trunc from 0x27f8df to 0x9e67c0 (0x766ee2 bytes) 000242[0] 1788869910.345742 mapwrite 0x870000 thru 0x87d582 (0xd583 bytes) 000243[1] 1788869910.596735 write_OD 0x83a000 thru 0x840fff (0x7000 bytes) 000247[1] 1788869910.746805 mapwrite 0x580000 thru 0x58862c (0x862d bytes) 000249[0] 1788869911.076262 trunc from 0x9e67c1 to 0x5c9c14 (0x41cbac bytes) 000250[1] 1788869911.269856 mapwrite 0x6b0000 thru 0x6bdc83 (0xdc84 bytes) 000251[1] 1788869911.441323 read_OD 0x5d1000 thru 0x5d2fff (0x2000 bytes) 000257[0] 1788869911.467773 read 0x138000 thru 0x13d6c0 (0x56c1 bytes) 000258[0] 1788869911.476522 trunc from 0x6bdc84 to 0x294042 (0x429c41 bytes) 000259[0] 1788869911.548520 mapread 0x0ed000 thru 0x0f28f4 (0x58f5 bytes) 000260[0] 1788869911.617994 mapread 0x28e000 thru 0x29347e (0x547f bytes) 000265[0] 1788869911.686786 mapread 0x059000 thru 0x05c488 (0x3489 bytes) 000267[0] 1788869911.716161 trunc from 0x294043 to 0x8691e1 (0x5d519f bytes) 000271[1] 1788869911.752576 write_OD 0x6ba000 thru 0x6bbfff (0x2000 bytes) 000273[0] 1788869911.806636 read 0x324000 thru 0x328b1c (0x4b1d bytes) 000275[0] 1788869911.860935 write 0x63d000 thru 0x6452be (0x82bf bytes) 000278[1] 1788869911.970171 write_OD 0x04c000 thru 0x04dfff (0x2000 bytes) 000280[1] 1788869912.044402 mapread 0x11a000 thru 0x1259e9 (0xb9ea bytes) 000281[1] 1788869912.086452 mapwrite 0x732000 thru 0x73b10d (0x910e bytes) 000283[1] 1788869912.189987 mapread 0x5dd000 thru 0x5ea75c (0xd75d bytes) 000284[0] 1788869912.217547 read 0x231000 thru 0x239ce9 (0x8cea bytes) 000285[1] 1788869912.241430 trunc from 0x8691e2 to 0x0ef503 (0x779cde bytes) 000286[0] 1788869912.271913 read 0x06f000 thru 0x06f7d8 (0x7d9 bytes) 000287[1] 1788869912.315185 write_OD 0x360000 thru 0x369fff (0xa000 bytes) 000288[0] 1788869912.459186 mapwrite 0x87a000 thru 0x87ac64 (0xc65 bytes) 000289[1] 1788869912.597376 read_OD 0x7c4000 thru 0x7c8fff (0x5000 bytes) 000294[1] 1788869912.654262 trunc from 0x87ac65 to 0x15e4ec (0x71c778 bytes) 000295[0] 1788869912.707731 mapwrite 0x3a0000 thru 0x3a55c1 (0x55c2 bytes) 000296[1] 1788869912.865107 read_OD 0x023000 thru 0x02ffff (0xd000 bytes) 000297[1] 1788869912.936032 mapwrite 0x00e000 thru 0x01026e (0x226f bytes) 000299[1] 1788869913.131545 read_OD 0x0b6000 thru 0x0bffff (0xa000 bytes) 000303[0] 1788869913.170388 mapwrite 0x1ab000 thru 0x1b0ff7 (0x5ff8 bytes) 000306[1] 1788869913.272486 write_OD 0x80b000 thru 0x818fff (0xe000 bytes) 000307[0] 1788869913.350354 write 0x9a4000 thru 0x9a68f5 (0x28f6 bytes) 000309[0] 1788869913.367281 write 0x647000 thru 0x653f79 (0xcf7a bytes) 000312[1] 1788869913.468722 mapread 0x12d000 thru 0x13037d (0x337e bytes) 000315[1] 1788869913.569814 mapwrite 0x4c7000 thru 0x4d1711 (0xa712 bytes) 000320[0] 1788869913.770694 mapwrite 0x320000 thru 0x327fbc (0x7fbd bytes) 000322[1] 1788869913.980735 write_OD 0x8e9000 thru 0x8f0fff (0x8000 bytes) 000323[1] 1788869914.070299 mapwrite 0x9ac000 thru 0x9ba104 (0xe105 bytes) 000324[1] 1788869914.288075 read_OD 0x314000 thru 0x31afff (0x7000 bytes) 000326[1] 1788869914.358330 read_OD 0x4f2000 thru 0x4f4fff (0x3000 bytes) 000330[0] 1788869914.366147 write 0x6e2000 thru 0x6e94de (0x74df bytes) 000331[1] 1788869914.385254 mapwrite 0x954000 thru 0x961fb5 (0xdfb6 bytes) 000335[0] 1788869914.539378 mapwrite 0x8c7000 thru 0x8d5041 (0xe042 bytes) 000336[1] 1788869914.667896 mapwrite 0x279000 thru 0x2845a4 (0xb5a5 bytes) 000337[0] 1788869914.771457 write 0x3c7000 thru 0x3cfe6a (0x8e6b bytes) 000339[0] 1788869914.795349 trunc from 0x9ba105 to 0x9a9364 (0x10da0 bytes) 000340[0] 1788869914.816627 read 0x696000 thru 0x6a126d (0xb26e bytes) 000341[1] 1788869914.829415 read_OD 0x659000 thru 0x662fff (0xa000 bytes) 000342[1] 1788869914.912316 write_OD 0x6b1000 thru 0x6b8fff (0x8000 bytes) 000347[1] 1788869915.087022 trunc from 0x9a9365 to 0x86e3b6 (0x13afae bytes) 000348[0] 1788869915.181316 mapread 0x53d000 thru 0x546958 (0x9959 bytes) 000350[0] 1788869915.268260 mapwrite 0x2e6000 thru 0x2f104d (0xb04e bytes) 000352[1] 1788869915.462904 read_OD 0x507000 thru 0x50cfff (0x6000 bytes) 000354[0] 1788869915.510886 mapwrite 0x69e000 thru 0x6add0c (0xfd0d bytes) 000357[1] 1788869915.630997 mapread 0x13c000 thru 0x14446d (0x846e bytes) 000362[0] 1788869915.671534 mapread 0x146000 thru 0x15092e (0xa92f bytes) 000363[1] 1788869915.688672 mapread 0x767000 thru 0x77368a (0xc68b bytes) 000365[0] 1788869915.713403 trunc from 0x86e3b7 to 0x91a1b4 (0xabdfe bytes) 000366[0] 1788869915.742497 read 0x701000 thru 0x70e16d (0xd16e bytes) 000367[1] 1788869915.750441 mapwrite 0x6a5000 thru 0x6a7e6d (0x2e6e bytes) 000369[0] 1788869915.836156 trunc from 0x91a1b5 to 0x62bf84 (0x2ee230 bytes) 000370[0] 1788869915.859297 read 0x4f5000 thru 0x5015c8 (0xc5c9 bytes) 000371[1] 1788869915.867021 read_OD 0x60f000 thru 0x61cfff (0xe000 bytes) 000373[0] 1788869915.901949 read 0x28b000 thru 0x29216a (0x716b bytes) 000375[0] 1788869915.911702 write 0x045000 thru 0x0547bd (0xf7be bytes) 000377[0] 1788869915.957588 read 0x3eb000 thru 0x3f776f (0xc770 bytes) 000379[0] 1788869915.967541 mapread 0x213000 thru 0x215d3f (0x2d40 bytes) 000383[1] 1788869916.068022 write_OD 0x8d1000 thru 0x8d8fff (0x8000 bytes) 000385[1] 1788869916.160551 mapread 0x7e3000 thru 0x7ee3e9 (0xb3ea bytes) 000387[0] 1788869916.213479 mapread 0x046000 thru 0x053c89 (0xdc8a bytes) 000388[0] 1788869916.256276 read 0x190000 thru 0x19e1f3 (0xe1f4 bytes) 000390[1] 1788869916.269774 trunc from 0x8d9000 to 0x715c2d (0x1c33d2 bytes) 000392[0] 1788869916.420433 read 0x2c1000 thru 0x2c7e00 (0x6e01 bytes) 000394[1] 1788869916.446925 read_OD 0x20a000 thru 0x20ffff (0x6000 bytes) 000396[1] 1788869916.465116 read_OD 0x2e5000 thru 0x2e5fff (0x1000 bytes) 000398[0] 1788869916.470457 read 0x553000 thru 0x55b738 (0x8739 bytes) 000400[0] 1788869916.481360 mapwrite 0x73c000 thru 0x73d465 (0x1466 bytes) 000401[1] 1788869916.633159 write_OD 0x9aa000 thru 0x9b2fff (0x9000 bytes) 000402[0] 1788869916.812847 trunc from 0x9b3000 to 0x14e86e (0x864791 bytes) 000403[1] 1788869916.863749 mapwrite 0x203000 thru 0x203e2d (0xe2e bytes) 000409[1] 1788869917.043046 mapwrite 0x2c8000 thru 0x2d597a (0xd97b bytes) 000411[0] 1788869917.168546 write 0x137000 thru 0x146f28 (0xff29 bytes) 000414[1] 1788869917.200918 trunc from 0x2d597b to 0x85e711 (0x588d97 bytes) 000416[1] 1788869917.268430 trunc from 0x85e712 to 0x6dfae8 (0x17ec29 bytes) 000418[0] 1788869917.309450 read 0x545000 thru 0x5457c5 (0x7c6 bytes) 000419[0] 1788869917.330405 mapread 0x32f000 thru 0x333ffb (0x4ffc bytes) 000420[0] 1788869917.349769 read 0x177000 thru 0x185ac6 (0xeac7 bytes) 000423[1] 1788869917.359922 write_OD 0x890000 thru 0x89cfff (0xd000 bytes) 000426[0] 1788869917.442423 mapread 0x213000 thru 0x218bad (0x5bae bytes) 000427[1] 1788869917.469534 write_OD 0x5a4000 thru 0x5a5fff (0x2000 bytes) 000432[0] 1788869917.589400 mapwrite 0x128000 thru 0x132e19 (0xae1a bytes) 000436[0] 1788869917.847388 trunc from 0x89d000 to 0x1412d2 (0x75bd2d bytes) 000438[1] 1788869917.945931 trunc from 0x1412d3 to 0x918a2c (0x7d775a bytes) 000441[1] 1788869918.029608 mapread 0x1fd000 thru 0x20844f (0xb450 bytes) 000444[0] 1788869918.089474 write 0x45b000 thru 0x461a72 (0x6a73 bytes) 000445[0] 1788869918.114019 write 0x3d9000 thru 0x3dbdb4 (0x2db5 bytes) 000449[0] 1788869918.151025 mapread 0x214000 thru 0x222bb5 (0xebb6 bytes) 000450[0] 1788869918.181047 mapread 0x354000 thru 0x355ed7 (0x1ed8 bytes) 000451[0] 1788869918.208118 mapread 0x74e000 thru 0x757db0 (0x9db1 bytes) 000452[0] 1788869918.229178 trunc from 0x918a2d to 0x4dfe1f (0x438c0d bytes) 000453[1] 1788869918.274984 trunc from 0x4dfe20 to 0x2ec328 (0x1f3af7 bytes) 000454[0] 1788869918.359494 write 0x2b7000 thru 0x2c42d7 (0xd2d8 bytes) 000456[0] 1788869918.406506 mapread 0x05d000 thru 0x05f8a0 (0x28a1 bytes) 000459[1] 1788869918.447447 mapwrite 0x85d000 thru 0x869c02 (0xcc03 bytes) 000462[0] 1788869918.719683 mapread 0x7b6000 thru 0x7ba8c7 (0x48c8 bytes) 000463[1] 1788869918.797351 read_OD 0x38e000 thru 0x39cfff (0xf000 bytes) 000466[0] 1788869918.821167 mapread 0x747000 thru 0x74fa8a (0x8a8b bytes) 000467[0] 1788869918.830134 read 0x0d5000 thru 0x0e1126 (0xc127 bytes) 000470[1] 1788869918.847629 mapwrite 0x45b000 thru 0x467276 (0xc277 bytes) 000471[0] 1788869919.058106 mapread 0x6e4000 thru 0x6eae90 (0x6e91 bytes) 000474[1] 1788869919.187269 trunc from 0x869c03 to 0x7f806e (0x71b94 bytes) 000477[0] 1788869919.397381 write 0x0ab000 thru 0x0ae2e5 (0x32e6 bytes) 000478[0] 1788869919.487116 trunc from 0x7f806f to 0x288c81 (0x56f3ed bytes) 000480[1] 1788869919.553729 mapread 0x275000 thru 0x2800c4 (0xb0c5 bytes) 000484[1] 1788869919.678574 mapwrite 0x563000 thru 0x568cf2 (0x5cf3 bytes) 000488[1] 1788869919.784252 trunc from 0x568cf3 to 0x14c7a3 (0x41c54f bytes) 000491[1] 1788869919.840610 mapread 0x011000 thru 0x01a7a5 (0x97a6 bytes) 000493[0] 1788869919.937075 trunc from 0x14c7a4 to 0x58f003 (0x442860 bytes) 000497[1] 1788869920.014156 trunc from 0x58f004 to 0x405fc2 (0x189041 bytes) All operations completed A-OK! PASS 16a (56s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 08:18:46 (1788869926) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.498579 s, 16.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.30237 s, 27.7 MB/s Chance of close/open is 1 in 50 Seed set to 9929 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1788869930.249056 write 0x51c2ae thru 0x51d794 (0x14e7 bytes) 000003[1] 1788869930.280072 write 0x4b531e thru 0x4b597e (0x661 bytes) 000004[0] 1788869930.313579 read 0x2307bf thru 0x23099b (0x1dd bytes) 000005[0] 1788869930.370445 mapread 0x0dc4b5 thru 0x0de2ef (0x1e3b bytes) 000006[0] 1788869930.420599 write 0x55480b thru 0x554d52 (0x548 bytes) 000007[1] 1788869930.483917 mapread 0x49071b thru 0x492677 (0x1f5d bytes) 000009[1] 1788869930.525649 read 0x4d9ee7 thru 0x4dbbc6 (0x1ce0 bytes) 000010[1] 1788869930.533017 mapwrite 0x5aa04b thru 0x5aa8c9 (0x87f bytes) 000011[0] 1788869930.655473 read 0x378bee thru 0x3793ea (0x7fd bytes) 000012[1] 1788869930.711762 write 0x0e92a3 thru 0x0ea0f9 (0xe57 bytes) 000014[0] 1788869930.786627 mapwrite 0x093e7f thru 0x094705 (0x887 bytes) 000016[1] 1788869930.931578 read 0x3fe048 thru 0x3fff19 (0x1ed2 bytes) 000017[0] 1788869930.950988 mapwrite 0x039af3 thru 0x03b091 (0x159f bytes) truncating to largest ever: 0x844f72 000022[0] 1788869931.195033 trunc from 0x5aa8ca to 0x844f71 (0x29a6a8 bytes) 000024[0] 1788869931.292655 mapread 0x44c420 thru 0x44c95f (0x540 bytes) 000029[0] 1788869931.355342 read 0x7d7a73 thru 0x7d99dc (0x1f6a bytes) 000030[0] 1788869931.373463 trunc from 0x844f72 to 0x411ac3 (0x4334ae bytes) 000032[1] 1788869931.475215 mapwrite 0x519adc thru 0x51b49b (0x19c0 bytes) 000033[1] 1788869931.621396 trunc from 0x51b49c to 0x100a85 (0x41aa16 bytes) 000035[0] 1788869931.674009 mapread 0x071769 thru 0x07243c (0xcd4 bytes) 000036[1] 1788869931.716578 trunc from 0x100a86 to 0x5170d6 (0x416651 bytes) 000037[1] 1788869931.750015 read 0x34bdc2 thru 0x34c01b (0x25a bytes) 000041[1] 1788869931.853949 read 0x287d72 thru 0x289d68 (0x1ff7 bytes) 000042[1] 1788869931.858578 write 0x7aaac1 thru 0x7ac569 (0x1aa9 bytes) 000043[0] 1788869931.882095 trunc from 0x7ac56a to 0x566d85 (0x2457e4 bytes) 000047[1] 1788869931.986428 mapread 0x248193 thru 0x248eb4 (0xd22 bytes) 000051[1] 1788869932.066528 write 0x4c748e thru 0x4c783b (0x3ae bytes) 000053[1] 1788869932.094403 read 0x1895e1 thru 0x1898ec (0x30c bytes) 000054[1] 1788869932.100356 trunc from 0x566d86 to 0x6190fb (0xb2376 bytes) 000055[1] 1788869932.149339 read 0x5b79cb thru 0x5b8b34 (0x116a bytes) 000059[1] 1788869932.202376 trunc from 0x6190fc to 0x4a9af8 (0x16f603 bytes) 000063[0] 1788869932.321398 read 0x2eb975 thru 0x2ec99c (0x1028 bytes) 000064[0] 1788869932.370767 mapwrite 0x21271c thru 0x2136bf (0xfa4 bytes) 000065[1] 1788869932.480979 mapread 0x1ff9b8 thru 0x200215 (0x85e bytes) 000067[0] 1788869932.523427 trunc from 0x4a9af9 to 0x411ac0 (0x98038 bytes) 000068[0] 1788869932.571403 read 0x36b80b thru 0x36c85a (0x1050 bytes) 000070[1] 1788869932.613572 write 0x15e4b3 thru 0x15f938 (0x1486 bytes) 000071[0] 1788869932.646938 trunc from 0x411ac1 to 0x5c741a (0x1b595a bytes) 000076[1] 1788869932.759453 mapwrite 0x91a6e7 thru 0x91c120 (0x1a3a bytes) 000077[1] 1788869932.857530 trunc from 0x91c121 to 0x314dcd (0x607353 bytes) 000078[0] 1788869932.886430 read 0x22bdf0 thru 0x22cca5 (0xeb6 bytes) 000080[0] 1788869932.977568 mapread 0x24a45c thru 0x24b083 (0xc28 bytes) 000083[1] 1788869932.985461 mapread 0x0fe153 thru 0x0ff432 (0x12e0 bytes) 000084[0] 1788869933.005267 write 0x5abfee thru 0x5ad351 (0x1364 bytes) 000086[0] 1788869933.030222 mapread 0x26414e thru 0x265751 (0x1604 bytes) 000089[0] 1788869933.054557 mapwrite 0x20ad52 thru 0x20c379 (0x1628 bytes) 000091[0] 1788869933.196949 read 0x59948c thru 0x59a03a (0xbaf bytes) 000094[1] 1788869933.254322 trunc from 0x5ad352 to 0x2479cc (0x365985 bytes) 000095[1] 1788869933.311604 mapread 0x0bd8fc thru 0x0bf602 (0x1d07 bytes) 000098[0] 1788869933.398688 mapread 0x0e884d thru 0x0ea84a (0x1ffe bytes) 000100[0] 1788869933.565830 mapread 0x14fc7d thru 0x15037a (0x6fe bytes) 000101[0] 1788869933.599325 read 0x075ab3 thru 0x076e16 (0x1364 bytes) 000104[0] 1788869933.704461 mapread 0x2214ee thru 0x2217e1 (0x2f4 bytes) 000107[1] 1788869933.790697 mapwrite 0x7b61bc thru 0x7b7480 (0x12c5 bytes) 000108[0] 1788869933.945116 mapwrite 0x53fe28 thru 0x53ff75 (0x14e bytes) 000109[0] 1788869934.108001 read 0x5c6bdf thru 0x5c70be (0x4e0 bytes) 000110[0] 1788869934.129637 write 0x47e9c0 thru 0x48025f (0x18a0 bytes) 000114[1] 1788869934.207532 mapread 0x41c173 thru 0x41d515 (0x13a3 bytes) 000119[0] 1788869934.300154 trunc from 0x7b7481 to 0x3c97a1 (0x3edcdf bytes) 000120[0] 1788869934.392164 mapwrite 0x7704e1 thru 0x7711a1 (0xcc1 bytes) truncating to largest ever: 0x9f2a14 000123[0] 1788869934.590620 trunc from 0x7711a2 to 0x9f2a13 (0x281872 bytes) 000125[1] 1788869934.758182 mapwrite 0x08792c thru 0x088c55 (0x132a bytes) 000128[1] 1788869934.857116 write 0x680afc thru 0x68265f (0x1b64 bytes) 000129[1] 1788869934.905141 mapwrite 0x9ed63f thru 0x9ed8a8 (0x26a bytes) 000132[0] 1788869934.994086 write 0x5c7a75 thru 0x5c816a (0x6f6 bytes) 000135[1] 1788869935.093332 write 0x626ad9 thru 0x6286a2 (0x1bca bytes) 000136[0] 1788869935.152545 trunc from 0x9f2a14 to 0x5b73f3 (0x43b620 bytes) 000138[1] 1788869935.233711 mapwrite 0x0ddaec thru 0x0deba5 (0x10ba bytes) 000142[1] 1788869935.407620 mapread 0x022004 thru 0x02299b (0x998 bytes) 000143[0] 1788869935.442643 trunc from 0x5b73f4 to 0x492e02 (0x1245f1 bytes) 000144[0] 1788869935.492637 trunc from 0x492e03 to 0x4cd39e (0x3a59c bytes) 000145[1] 1788869935.535101 mapwrite 0x8363ba thru 0x837ffa (0x1c41 bytes) 000146[1] 1788869935.632618 trunc from 0x837ffb to 0x5fae5b (0x23d19f bytes) 000147[0] 1788869935.660332 mapwrite 0x4ac872 thru 0x4ad09a (0x829 bytes) 000149[0] 1788869935.780762 write 0x8e38ee thru 0x8e4793 (0xea6 bytes) 000150[1] 1788869935.805095 mapwrite 0x767a26 thru 0x768d86 (0x1361 bytes) 000151[1] 1788869935.977383 read 0x720453 thru 0x721d58 (0x1906 bytes) 000155[1] 1788869936.063439 write 0x4f8083 thru 0x4f9348 (0x12c6 bytes) 000159[1] 1788869936.235408 trunc from 0x8e4794 to 0x844542 (0xa0251 bytes) 000162[0] 1788869936.324605 mapwrite 0x6ba80a thru 0x6bb980 (0x1177 bytes) 000164[1] 1788869936.432384 write 0x40d55e thru 0x40e734 (0x11d7 bytes) 000167[0] 1788869936.503536 mapwrite 0x616b9b thru 0x617de7 (0x124d bytes) 000171[1] 1788869936.647312 read 0x57a1fc thru 0x57b373 (0x1178 bytes) 000172[0] 1788869936.679515 write 0x2c6cee thru 0x2c841e (0x1731 bytes) 000178[1] 1788869936.752549 mapwrite 0x9c7b2a thru 0x9c8d82 (0x1259 bytes) 000180[1] 1788869936.961467 mapwrite 0x3569d5 thru 0x356f7a (0x5a6 bytes) 000181[1] 1788869937.068440 read 0x5758d7 thru 0x575cbb (0x3e5 bytes) 000182[1] 1788869937.102569 trunc from 0x9c8d83 to 0x0c88fc (0x900486 bytes) 000184[1] 1788869937.173509 write 0x469504 thru 0x46b2c0 (0x1dbd bytes) 000185[1] 1788869937.179044 read 0x000822 thru 0x0016a6 (0xe85 bytes) 000186[0] 1788869937.196159 mapwrite 0x9023b5 thru 0x903217 (0xe63 bytes) 000189[0] 1788869937.367009 mapread 0x00cf75 thru 0x00db1e (0xbaa bytes) 000190[1] 1788869937.399688 read 0x6e4f0d thru 0x6e619c (0x1290 bytes) 000192[1] 1788869937.472620 trunc from 0x903218 to 0x58dd62 (0x3754b5 bytes) 000195[1] 1788869937.647395 mapread 0x255a3d thru 0x2561ad (0x771 bytes) 000198[0] 1788869937.797999 read 0x5635b0 thru 0x563bb5 (0x606 bytes) 000199[1] 1788869937.855318 mapwrite 0x256f6f thru 0x25881f (0x18b1 bytes) 000200[1] 1788869937.965805 mapread 0x0ccc8b thru 0x0cd04a (0x3c0 bytes) 000205[1] 1788869938.038610 mapwrite 0x1ed3e8 thru 0x1ee9fe (0x1617 bytes) 000206[0] 1788869938.166172 read 0x1d31f8 thru 0x1d4a80 (0x1889 bytes) 000208[1] 1788869938.207620 write 0x2a6231 thru 0x2a78a5 (0x1675 bytes) 000210[0] 1788869938.257065 write 0x1f88f6 thru 0x1fa1bd (0x18c8 bytes) 000214[1] 1788869938.292611 trunc from 0x58dd63 to 0x3d96cd (0x1b4695 bytes) 000215[1] 1788869938.338029 mapread 0x249103 thru 0x249de1 (0xcdf bytes) 000217[1] 1788869938.381646 read 0x19770b thru 0x198dfd (0x16f3 bytes) 000218[0] 1788869938.432953 mapwrite 0x06c8c0 thru 0x06e260 (0x19a1 bytes) 000220[1] 1788869938.568295 mapwrite 0x613a16 thru 0x61555a (0x1b45 bytes) 000221[0] 1788869938.855690 mapread 0x48b8b3 thru 0x48bd23 (0x471 bytes) 000223[1] 1788869938.969663 write 0x18a56f thru 0x18b219 (0xcab bytes) 000224[1] 1788869938.985174 read 0x0e0d60 thru 0x0e1718 (0x9b9 bytes) 000225[1] 1788869938.992079 write 0x47603b thru 0x47632e (0x2f4 bytes) 000226[0] 1788869939.007151 mapread 0x0bbf89 thru 0x0bc11e (0x196 bytes) 000230[0] 1788869939.044361 write 0x897d04 thru 0x8994a0 (0x179d bytes) 000232[1] 1788869939.087876 read 0x59820d thru 0x59879a (0x58e bytes) 000233[0] 1788869939.103316 trunc from 0x8994a1 to 0x16fc5b (0x729845 bytes) 000235[1] 1788869939.152220 write 0x43172a thru 0x432f61 (0x1838 bytes) 000236[1] 1788869939.175377 write 0x2c8ab5 thru 0x2c9358 (0x8a4 bytes) 000237[0] 1788869939.211600 read 0x3e91cd thru 0x3e9b24 (0x958 bytes) 000238[0] 1788869939.259359 trunc from 0x432f62 to 0x95540e (0x5224ad bytes) 000239[1] 1788869939.326140 mapwrite 0x57744f thru 0x57915b (0x1d0d bytes) 000241[1] 1788869939.441040 mapwrite 0x87ab04 thru 0x87c9ef (0x1eec bytes) 000242[1] 1788869939.653418 mapread 0x002438 thru 0x002f8b (0xb54 bytes) 000244[1] 1788869939.722103 mapread 0x37674a thru 0x37693a (0x1f1 bytes) 000247[1] 1788869939.730169 write 0x1dd915 thru 0x1ddf62 (0x64e bytes) 000249[1] 1788869939.783609 mapwrite 0x8d7e7f thru 0x8d8c81 (0xe03 bytes) 000251[1] 1788869939.951899 mapwrite 0x5be695 thru 0x5c04be (0x1e2a bytes) 000254[1] 1788869940.185253 mapread 0x5d720c thru 0x5d8aae (0x18a3 bytes) 000255[0] 1788869940.241472 mapwrite 0x6add67 thru 0x6ae659 (0x8f3 bytes) 000258[0] 1788869940.457292 mapwrite 0x47468b thru 0x474ca7 (0x61d bytes) 000267[1] 1788869940.817812 mapwrite 0x4843c9 thru 0x4853bc (0xff4 bytes) 000268[1] 1788869940.976149 trunc from 0x95540f to 0x77d925 (0x1d7ae9 bytes) 000272[0] 1788869941.080836 trunc from 0x77d926 to 0x19a127 (0x5e37fe bytes) 000273[1] 1788869941.141832 trunc from 0x19a128 to 0x239898 (0x9f771 bytes) 000275[0] 1788869941.177543 write 0x219648 thru 0x21b549 (0x1f02 bytes) 000279[0] 1788869941.294178 mapread 0x1a97ac thru 0x1ab191 (0x19e6 bytes) 000280[1] 1788869941.338556 write 0x5e3b32 thru 0x5e49f7 (0xec6 bytes) 000281[1] 1788869941.352124 mapwrite 0x91f3d6 thru 0x91f451 (0x7c bytes) 000282[0] 1788869941.431143 read 0x6f20ff thru 0x6f3aa8 (0x19aa bytes) 000283[1] 1788869941.458914 mapwrite 0x5a93a4 thru 0x5aa103 (0xd60 bytes) 000284[1] 1788869941.619067 mapwrite 0x80d0a3 thru 0x80e897 (0x17f5 bytes) 000287[0] 1788869941.731320 mapwrite 0x5d9858 thru 0x5db4a2 (0x1c4b bytes) 000288[1] 1788869941.893848 write 0x6d7c32 thru 0x6d8681 (0xa50 bytes) 000290[1] 1788869941.953664 mapread 0x8af4ef thru 0x8afe85 (0x997 bytes) 000291[1] 1788869942.014985 mapwrite 0x2aa392 thru 0x2ab02b (0xc9a bytes) 000292[1] 1788869942.211556 trunc from 0x91f452 to 0x03c838 (0x8e2c19 bytes) 000293[1] 1788869942.304443 read 0x0151e7 thru 0x016f7b (0x1d95 bytes) 000294[0] 1788869942.338543 trunc from 0x03c839 to 0x72b545 (0x6eed0d bytes) 000299[0] 1788869942.498674 mapread 0x6218f6 thru 0x622357 (0xa62 bytes) 000300[1] 1788869942.645437 trunc from 0x72b546 to 0x88f00f (0x163aca bytes) 000302[0] 1788869942.698162 mapread 0x44b0ce thru 0x44c343 (0x1276 bytes) 000303[0] 1788869942.760185 mapread 0x34f017 thru 0x350032 (0x101c bytes) 000304[0] 1788869942.801441 trunc from 0x88f010 to 0x20d1e3 (0x681e2c bytes) 000306[1] 1788869942.874312 mapread 0x1797f4 thru 0x179b19 (0x326 bytes) 000307[0] 1788869942.962567 read 0x0d043a thru 0x0d0c46 (0x80d bytes) 000308[0] 1788869943.009774 read 0x10f6a6 thru 0x10fe3c (0x797 bytes) 000309[1] 1788869943.015553 mapread 0x0c1980 thru 0x0c38c8 (0x1f49 bytes) 000310[1] 1788869943.039596 read 0x19e0f3 thru 0x19ff93 (0x1ea1 bytes) 000311[0] 1788869943.062419 trunc from 0x20d1e4 to 0x2b80e7 (0xaaf04 bytes) 000312[0] 1788869943.171526 mapread 0x232423 thru 0x23366c (0x124a bytes) 000314[1] 1788869943.265504 mapread 0x05ac8f thru 0x05b6ec (0xa5e bytes) 000315[1] 1788869943.444373 mapread 0x0345d2 thru 0x0351d0 (0xbff bytes) 000316[0] 1788869943.499711 read 0x239304 thru 0x23b0b8 (0x1db5 bytes) 000317[0] 1788869943.661354 read 0x1fbb11 thru 0x1fd0e1 (0x15d1 bytes) 000318[0] 1788869943.666802 write 0x98ceea thru 0x98e7a0 (0x18b7 bytes) 000319[0] 1788869943.748230 trunc from 0x98e7a1 to 0x1b4983 (0x7d9e1d bytes) 000321[1] 1788869943.837309 read 0x0ad996 thru 0x0ae4ea (0xb55 bytes) 000327[1] 1788869944.163324 trunc from 0x1b4984 to 0x3223c3 (0x16da40 bytes) 000329[0] 1788869944.254902 mapread 0x1e0c85 thru 0x1e2771 (0x1aed bytes) 000330[1] 1788869944.389324 mapwrite 0x41834d thru 0x418911 (0x5c5 bytes) 000332[1] 1788869944.576899 trunc from 0x418912 to 0x5de74a (0x1c5e39 bytes) 000334[0] 1788869944.618490 read 0x3e0199 thru 0x3e1103 (0xf6b bytes) 000338[1] 1788869944.706390 write 0x64e7d2 thru 0x64ef5f (0x78e bytes) 000343[0] 1788869944.759863 mapread 0x048480 thru 0x0488b8 (0x439 bytes) 000350[0] 1788869944.814531 write 0x2c82ce thru 0x2c9324 (0x1057 bytes) 000351[0] 1788869944.859314 read 0x20ff8c thru 0x210efb (0xf70 bytes) 000352[1] 1788869944.885097 mapread 0x34b06d thru 0x34bc54 (0xbe8 bytes) 000356[0] 1788869945.018615 write 0x4a0752 thru 0x4a1833 (0x10e2 bytes) 000358[1] 1788869945.061892 read 0x5a67af thru 0x5a7cf8 (0x154a bytes) 000359[0] 1788869945.107552 trunc from 0x64ef60 to 0x628ea2 (0x260bd bytes) 000361[0] 1788869945.221807 write 0x24d861 thru 0x24eaa6 (0x1246 bytes) 000364[1] 1788869945.285174 mapread 0x0da064 thru 0x0da1da (0x177 bytes) 000365[0] 1788869945.329521 write 0x6e4408 thru 0x6e5f57 (0x1b50 bytes) 000367[0] 1788869945.350414 read 0x248a11 thru 0x2495a1 (0xb91 bytes) 000368[0] 1788869945.356181 trunc from 0x6e5f58 to 0x2de810 (0x407747 bytes) 000369[1] 1788869945.393530 write 0x8d9dcf thru 0x8db23b (0x146d bytes) 000371[1] 1788869945.440352 trunc from 0x8db23c to 0x7db252 (0xfffe9 bytes) 000373[0] 1788869945.481592 mapwrite 0x6331e1 thru 0x633d94 (0xbb4 bytes) 000374[0] 1788869945.637434 read 0x565ece thru 0x5676c4 (0x17f7 bytes) 000378[1] 1788869945.850280 write 0x46abcc thru 0x46b2ab (0x6e0 bytes) 000380[1] 1788869945.932539 mapread 0x61d50c thru 0x61dc0b (0x700 bytes) 000381[0] 1788869945.965152 mapread 0x50b2b3 thru 0x50cc9d (0x19eb bytes) 000382[1] 1788869946.020422 mapwrite 0x047c69 thru 0x0485aa (0x942 bytes) 000384[0] 1788869946.093429 mapwrite 0x8c0e48 thru 0x8c1934 (0xaed bytes) 000386[0] 1788869946.243350 trunc from 0x8c1935 to 0x3d5c27 (0x4ebd0d bytes) 000393[1] 1788869946.306558 trunc from 0x3d5c28 to 0x4d7c74 (0x10204d bytes) 000396[0] 1788869946.392593 mapread 0x37ef5f thru 0x3806d1 (0x1773 bytes) 000398[1] 1788869946.448787 read 0x1a0c82 thru 0x1a20fb (0x147a bytes) 000399[0] 1788869946.454175 mapread 0x100a6c thru 0x101452 (0x9e7 bytes) 000402[1] 1788869946.483601 mapwrite 0x11f4d5 thru 0x1207a9 (0x12d5 bytes) 000403[1] 1788869946.560229 mapread 0x1fcf77 thru 0x1fd855 (0x8df bytes) 000408[1] 1788869946.657779 write 0x608356 thru 0x608ca5 (0x950 bytes) 000410[0] 1788869946.694578 read 0x1b1711 thru 0x1b2c58 (0x1548 bytes) 000411[0] 1788869946.719848 mapwrite 0x04a705 thru 0x04c4b8 (0x1db4 bytes) 000415[0] 1788869946.944978 trunc from 0x608ca6 to 0x8ab3b9 (0x2a2714 bytes) 000416[1] 1788869947.073978 trunc from 0x8ab3ba to 0x9e3ebc (0x138b03 bytes) 000417[0] 1788869947.167721 read 0x883044 thru 0x88399c (0x959 bytes) 000418[1] 1788869947.190028 write 0x417615 thru 0x419234 (0x1c20 bytes) 000420[1] 1788869947.229225 write 0x8d8fc9 thru 0x8da1f3 (0x122b bytes) 000421[1] 1788869947.279666 write 0x8e407e thru 0x8e51e9 (0x116c bytes) 000424[0] 1788869947.310461 mapwrite 0x6d2aba thru 0x6d3a94 (0xfdb bytes) 000430[0] 1788869947.495981 mapread 0x55b180 thru 0x55c6e4 (0x1565 bytes) 000431[1] 1788869947.537780 read 0x904fcf thru 0x905327 (0x359 bytes) 000433[0] 1788869947.584845 read 0x0b91cf thru 0x0b9e7f (0xcb1 bytes) 000434[0] 1788869947.614891 write 0x803e35 thru 0x805cc7 (0x1e93 bytes) 000436[0] 1788869947.662537 read 0x1181ee thru 0x11a1c8 (0x1fdb bytes) 000440[1] 1788869947.690025 mapread 0x44e8b4 thru 0x4508a1 (0x1fee bytes) 000441[0] 1788869947.738883 mapwrite 0x2ac499 thru 0x2ad35e (0xec6 bytes) 000442[1] 1788869947.783887 trunc from 0x9e3ebd to 0x5250da (0x4bede2 bytes) 000448[0] 1788869947.956317 write 0x52c3f2 thru 0x52c885 (0x494 bytes) 000451[0] 1788869948.025378 read 0x15d86a thru 0x15d878 (0xf bytes) 000452[0] 1788869948.039173 mapwrite 0x8ffd43 thru 0x900298 (0x556 bytes) 000454[0] 1788869948.105903 read 0x7d5920 thru 0x7d674e (0xe2f bytes) 000455[0] 1788869948.131610 trunc from 0x900299 to 0x098df5 (0x8674a3 bytes) 000457[0] 1788869948.151247 read 0x021d3a thru 0x023c58 (0x1f1f bytes) 000459[0] 1788869948.237606 write 0x74e9ca thru 0x74ec2e (0x265 bytes) 000462[0] 1788869948.294297 mapread 0x641a4c thru 0x6438c4 (0x1e79 bytes) 000463[0] 1788869948.328127 trunc from 0x74ec2f to 0x554b93 (0x1fa09b bytes) 000464[1] 1788869948.365870 mapread 0x03e3d0 thru 0x03ffa5 (0x1bd6 bytes) 000465[0] 1788869948.455332 mapread 0x475151 thru 0x476caa (0x1b5a bytes) 000467[1] 1788869948.605082 mapwrite 0x025795 thru 0x026972 (0x11de bytes) 000468[0] 1788869948.713539 mapwrite 0x888e01 thru 0x88a94d (0x1b4d bytes) 000471[0] 1788869948.879382 read 0x357343 thru 0x357bf0 (0x8ae bytes) 000474[1] 1788869948.919316 mapread 0x152d86 thru 0x154a01 (0x1c7c bytes) 000475[1] 1788869948.968731 trunc from 0x88a94e to 0x3ee948 (0x49c005 bytes) 000481[0] 1788869949.111535 mapwrite 0x0a2b80 thru 0x0a3a44 (0xec5 bytes) 000482[1] 1788869949.225896 write 0x496ece thru 0x497e2b (0xf5e bytes) 000483[1] 1788869949.249532 trunc from 0x497e2c to 0x87cdaf (0x3e4f84 bytes) 000484[1] 1788869949.297502 write 0x5856d4 thru 0x5861cd (0xafa bytes) 000487[0] 1788869949.374962 write 0x69d718 thru 0x69e740 (0x1029 bytes) 000489[0] 1788869949.434081 mapwrite 0x7d604e thru 0x7d6621 (0x5d4 bytes) 000490[0] 1788869949.513791 trunc from 0x87cdb0 to 0x249514 (0x63389b bytes) 000492[0] 1788869949.654513 mapread 0x195fc9 thru 0x197537 (0x156f bytes) 000494[0] 1788869949.708326 read 0x041ceb thru 0x042257 (0x56d bytes) 000497[0] 1788869949.840392 mapwrite 0x7bc1ec thru 0x7bc24e (0x63 bytes) 000498[1] 1788869950.032108 mapwrite 0x5735ba thru 0x5741b2 (0xbf9 bytes) 000499[0] 1788869950.159517 mapwrite 0x39cb5f thru 0x39de7a (0x131c bytes) All operations completed A-OK! PASS 16b (30s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 08:19:16 (1788869956) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 08:19:20 (1788869960) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.342872 s, 24.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.332631 s, 25.2 MB/s ldlm.namespaces.MGC192.168.203.159@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8e7d12746800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8e7d12dd3800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12746800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12746800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5376 s, 9.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.77514 s, 27.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.03797 s, 34.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.19527 s, 16.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.27711 s, 12.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.92196 s, 26.7 MB/s PASS 16d (50s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 08:20:11 (1788870011) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.365491 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124091 s, 8.5 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 08:20:18 (1788870018) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 08:20:45 (1788870045) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 08:21:13 (1788870073) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 08:21:20 (1788870080) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 08:21:27 (1788870087) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.47089 s, 10.6 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.2971 s, 20.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.68462 s, 9.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.80877 s, 14.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.42105 s, 10.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.99148 s, 13.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.39192 s, 11.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.27002 s, 11.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.33855 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.08457 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.28957 s, 8.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.33183 s, 11.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.45017 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.85304 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.03781 s, 8.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.79521 s, 6.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.17246 s, 6.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.26595 s, 6.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.37034 s, 6.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (40s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 08:22:06 (1788870126) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Sep 8 08:22:51 EDT 2026 + fsx 2 done: rc=0 Tue Sep 8 08:22:51 EDT 2026 + fsx 3 done: rc=0 Tue Sep 8 08:22:51 EDT 2026 + fsx 4 done: rc=0 Tue Sep 8 08:22:51 EDT 2026 + fsx 5 done: rc=0 Tue Sep 8 08:22:51 EDT 2026 PASS 16k (50s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 08:22:56 (1788870176) fail_loc=0x8000030a PASS 17 (9s) == sanityn test 18: mmap sanity check =========================================================================================== 08:23:05 (1788870185) mmap test1: basic mmap operation (PASS, 0.12634s) mmap test2: MAP_PRIVATE not write back (PASS, 0.10753s) mmap test3: concurrent mmap ops on two nodes (PASS, 10.458s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5173s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.19971s) mmap test6: check mmap write/read content on two nodes (PASS, 0.209s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.5522s) mmap test11: multi-thread shared mmap (PASS, 1.2416s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1186s) PASS 18 (28s) == sanityn test 19: test concurrent uncached read races ========================================================================= 08:23:33 (1788870213) oleg359-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 08:23:37 (1788870217) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 08:23:44 (1788870224) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 08:23:51 (1788870231) now is 1788870233 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 new atime is 1788870294 PASS 23 (69s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 08:25:00 (1788870300) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (7s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 08:25:08 (1788870308) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 08:25:14 (1788870314) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 08:25:21 (1788870321) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 08:25:25 (1788870325) utime: good mknod times 1788870326 <= 1788870327 <= 1788870327 for /mnt/lustre/f26a utime: good mknod times 1788870326 <= 1788870327 <= 1788870327 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 08:25:32 (1788870332) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 08:25:40 (1788870340) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502396 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-08 08:25:42.000000000 -0400 Birth: 2026-09-08 08:25:41.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502396 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-08 08:25:42.000000000 -0400 Birth: 2026-09-08 08:25:41.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 08:25:50 (1788870350) 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 08:25:58 (1788870358) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 08:26:06 (1788870366) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg359-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.159@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 08:26:27 (1788870387) -rw-r--r-- 1 root root 0 Sep 8 08:26 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 08:26:36 (1788870396) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43096 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 08:26:44 (1788870404) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 08:26:51 (1788870411) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 08:26:55 (1788870415) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.5257 s, 34.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.34979 s, 38.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.11692 s, 24.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.32589 s, 22.5 MB/s contention_seconds 4, contention_events 19 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.99819 s, 26.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.20653 s, 23.8 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.25672 s, 41.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.2849 s, 40.8 MB/s PASS 32c (28s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 08:27:25 (1788870445) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 08:27:28 (1788870448) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 08:27:31 (1788870451) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 08:27:34 (1788870454) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 08:27:37 (1788870457) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e7d12746800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e7d12746800.ost_server_uuid in IDLE state after 0 sec oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e7d12746800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e7d12746800.ost_server_uuid in FULL state after 0 sec oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e7d12746800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e7d12746800.ost_server_uuid in IDLE state after 0 sec oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e7d12746800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e7d12746800.ost_server_uuid in FULL state after 0 sec oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8e7d12746800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e7d12746800.ost_server_uuid in IDLE state after 0 sec oleg359-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e7d12746800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e7d12746800.ost_server_uuid in FULL state after 0 sec PASS 34 (132s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 08:29:49 (1788870589) fail_loc=0x80000317 Race attempt 0 Wait for 50399 50436 for 60 sec... - open/close 747 (time 1788870605.69 total 10.02 last 74.57) - open/close 1502 (time 1788870615.69 total 20.02 last 75.45) - open/close 2320 (time 1788870625.70 total 30.03 last 81.77) - open/close 3083 (time 1788870635.71 total 40.04 last 76.24) - open/close 3870 (time 1788870645.71 total 50.04 last 78.68) total: 4000 open/close in 51.44 seconds: 77.76 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.159@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.203.159@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 08:31:00 (1788870660) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg359-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.32727 s, 39.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43274 s, 36.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49069 s, 35.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7492608) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.22733 s, 42.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.4334 s, 36.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.06886 s, 49.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.46556 s, 35.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7494656) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.16495 s, 45.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29865 s, 40.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7492608) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.58418 s, 33.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48422 s, 35.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7494656) after_dd(7443456) after(7494656) PASS 36 (307s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 08:36:07 (1788870967) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7604 - create 1532 (time 1788870981.48 total 10.01 last 152.98) - create 2960 (time 1788870991.49 total 20.02 last 142.74) - create 4448 (time 1788871001.49 total 30.02 last 148.77) - create 5968 (time 1788871011.49 total 40.02 last 151.97) - create 7444 (time 1788871021.50 total 50.03 last 147.46) - create 8750 (time 1788871031.50 total 60.03 last 130.56) total: 10000 create in 69.06 seconds: 144.80 ops/second PASS 37 (80s) == sanityn test 39a: file mtime does not change after rename ========================================================== 08:37:27 (1788871047) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 08:37:34 (1788871054) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 08:37:43 (1788871063) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 08:37:51 (1788871071) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 08:37:59 (1788871079) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 08:38:16 (1788871096) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 08:38:32 (1788871112) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (17s) == sanityn test 40e: pdirops: rename and others ======================================================================== 08:38:49 (1788871129) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (15s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 08:39:05 (1788871145) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 08:39:17 (1788871157) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 08:39:29 (1788871169) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (11s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 08:39:40 (1788871180) Conflict PASS 41d (13s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 08:39:53 (1788871193) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 08:40:06 (1788871206) Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 08:40:19 (1788871219) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 08:40:32 (1788871232) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 08:40:45 (1788871245) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (979s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 08:57:04 (1788872224) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 08:57:17 (1788872237) open: File exists Conflict PASS 42b (12s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 08:57:29 (1788872249) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (12s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 08:57:42 (1788872262) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 08:57:54 (1788872274) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (12s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 08:58:06 (1788872286) Conflict PASS 42f (12s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 08:58:18 (1788872298) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 08:58:31 (1788872311) Conflict PASS 42h (13s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 08:58:44 (1788872324) PASS 43a (67s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 08:59:51 (1788872391) Conflict PASS 43b (12s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 09:00:03 (1788872403) Conflict PASS 43c (13s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 09:00:16 (1788872416) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (12s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 09:00:28 (1788872428) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 09:00:41 (1788872441) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (12s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 09:00:53 (1788872453) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 09:01:06 (1788872466) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 09:01:19 (1788872479) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 09:01:23 (1788872483) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (133s) == sanityn test 43k: unlink vs create ==================== 09:03:37 (1788872617) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1123s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 09:22:20 (1788873740) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Sep 8 09:22:29 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 09:22:35 (1788873755) open: File exists Conflict PASS 44b (13s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 09:22:48 (1788873768) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (13s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 09:23:01 (1788873781) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 09:23:14 (1788873794) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 09:23:28 (1788873808) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 09:23:42 (1788873822) Conflict PASS 44g (15s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 09:23:57 (1788873837) Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 09:24:11 (1788873851) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 09:24:15 (1788873855) PASS 45a (119s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 09:26:14 (1788873974) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 09:26:27 (1788873987) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 09:26:41 (1788874001) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (12s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 09:26:53 (1788874013) Conflict PASS 45e (12s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 09:27:05 (1788874025) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (13s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 09:27:18 (1788874038) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (12s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 09:27:31 (1788874051) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 09:27:43 (1788874063) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 09:27:47 (1788874067) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1165s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 09:47:12 (1788875232) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (15s) == sanityn test 46b: pdirops: link vs create ======================================================================== 09:47:27 (1788875247) open: File exists Conflict PASS 46b (15s) == sanityn test 46c: pdirops: link vs link ======================================================================== 09:47:42 (1788875262) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (14s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 09:47:57 (1788875277) Conflict PASS 46d (14s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 09:48:11 (1788875291) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 09:48:24 (1788875304) Conflict PASS 46f (14s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 09:48:38 (1788875318) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 09:48:51 (1788875331) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 09:49:05 (1788875345) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 09:49:09 (1788875349) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 09:49:12 (1788875352) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 09:49:16 (1788875356) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 09:49:19 (1788875359) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 09:49:23 (1788875363) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 09:49:27 (1788875367) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 09:49:30 (1788875370) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 09:49:34 (1788875374) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0971028 s, 105 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 09:49:46 (1788875386) 0+1 records in 0+1 records out 158 bytes copied, 0.0448727 s, 3.5 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218659) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 09:49:56 (1788875396) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0233706 s, 43.8 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993347 s, 10.6 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 09:50:16 (1788875416) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0373472 s, 27.4 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 09:50:29 (1788875429) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822607 s, 12.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 09:50:37 (1788875437) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 09:50:46 (1788875446) llite.lustre-ffff8e7d12746800.xattr_cache=1 llite.lustre-ffff8e7d12dd3800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505013 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 09:50:48.000000000 -0400 Modify: 2026-09-08 09:50:48.000000000 -0400 Change: 2026-09-08 09:50:48.000000000 -0400 Birth: 2026-09-08 09:50:48.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 09:50:55.000000000 -0400 Modify: 2026-09-08 09:50:56.000000000 -0400 Change: 2026-09-08 09:50:56.000000000 -0400 Birth: 2026-09-08 09:50:54.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 09:51:01.000000000 -0400 Modify: 2026-09-08 09:51:01.000000000 -0400 Change: 2026-09-08 09:51:01.000000000 -0400 Birth: 2026-09-08 09:51:00.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 09:51:01.000000000 -0400 Modify: 2026-09-08 09:51:02.000000000 -0400 Change: 2026-09-08 09:51:02.000000000 -0400 Birth: 2026-09-08 09:51:00.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 09:51:07.000000000 -0400 Modify: 2026-09-08 09:51:08.000000000 -0400 Change: 2026-09-08 09:51:08.000000000 -0400 Birth: 2026-09-08 09:51:07.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-08 09:51:07.000000000 -0400 Modify: 2026-09-08 09:51:08.000000000 -0400 Change: 2026-09-08 09:51:08.000000000 -0400 Birth: 2026-09-08 09:51:07.000000000 -0400 PASS 54 (33s) == sanityn test 55a: rename vs unlink target dir ========= 09:51:19 (1788875479) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 09:51:33 (1788875493) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 09:51:46 (1788875506) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 09:52:05 (1788875525) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (15s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 09:52:20 (1788875540) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 09:52:23 (1788875543) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 09:52:42 (1788875562) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (23s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 09:53:06 (1788875586) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788875588 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.592612 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (110s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 09:54:56 (1788875696) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788875700 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.17379 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (116s) == sanityn test 60: Verify data_version behaviour ======== 09:56:52 (1788875812) PASS 60 (7s) == sanityn test 70a: cd directory && rm directory ======== 09:56:59 (1788875819) PASS 70a (7s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 09:57:07 (1788875827) PASS 70b (7s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 09:57:14 (1788875834) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0195431 s, 3.4 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 09:57:18 (1788875838) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0128442 s, 5.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 09:57:21 (1788875841) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 09:57:25 (1788875845) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0269215 s, 2.4 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 09:57:28 (1788875848) llite.lustre-ffff8e7d12746800.xattr_cache=1 llite.lustre-ffff8e7d12dd3800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 09:57:35 (1788875855) llite.lustre-ffff8e7d12746800.xattr_cache=1 llite.lustre-ffff8e7d12dd3800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 09:57:42 (1788875862) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0183455 s, 55.8 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0106918 s, 95.8 kB/s 231481: get lock2 231482: get lock1 231481: done 231481: get lock1 231482: done 231482: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231482: cannot get lock2: Resource deadlock avoided 231482: exit rc=0 231481: done 231481: exit rc=0 PASS 74 (9s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 09:57:51 (1788875871) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0980086 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334046 s, 31.4 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232062 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (12s) == sanityn test 76: Verify MDT open_files listing ======== 09:58:03 (1788875883) opening files: ........ closing files: ........ PASS 76 (69s) == sanityn test 77a: check FIFO NRS policy =============== 09:59:13 (1788875953) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.401137 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550404 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383246 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109528 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115342 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075639 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134944 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184822 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119217 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223832 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270673 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232893 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199908 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219719 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212248 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235153 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274155 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846577 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289616 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438267 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575008 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0099603 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00223932 s, 468 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00275119 s, 381 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00250766 s, 418 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105388 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00195852 s, 535 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859427 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864094 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920988 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463725 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937881 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951891 s, 11.0 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (8s) == sanityn test 77b: check CRR-N NRS policy ============== 09:59:21 (1788875961) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.428245 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692878 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889214 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121338 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168228 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077328 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805431 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157371 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138131 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121748 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260167 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18654 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162671 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215651 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634036 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307742 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500275 s, 21.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552228 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166199 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119824 s, 87.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113047 s, 92.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051387 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528323 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480167 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570067 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367445 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045227 s, 232 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00331179 s, 317 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450816 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00851801 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00361953 s, 290 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114636 s, 91.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00771718 s, 136 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.545991 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049326 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112957 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756123 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154872 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100523 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188102 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168405 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129955 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20043 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752503 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237593 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13536 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219327 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136953 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147059 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305351 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562938 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534127 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196096 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014549 s, 72.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237675 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370861 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677238 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00966726 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330955 s, 31.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242493 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00738774 s, 142 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0096176 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130104 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00300917 s, 348 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232903 s, 45.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465746 s, 22.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (11s) == sanityn test 77c: check ORR NRS policy ================ 09:59:32 (1788875972) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.393194 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542796 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102906 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825093 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805204 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117573 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126972 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102227 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166951 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198639 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240859 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280701 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288551 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274871 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252983 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247212 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332469 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12898 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839343 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139627 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087237 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763622 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939141 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724848 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05687 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833264 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110117 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789398 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812848 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685064 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620158 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671319 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819308 s, 12.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.413672 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843524 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819453 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100406 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188619 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181595 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108811 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152359 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12417 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184303 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280914 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230855 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223922 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188369 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268453 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278593 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223698 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561557 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648953 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283865 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463163 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423058 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428293 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457736 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623504 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02629 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406197 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681912 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445612 s, 23.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637003 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065004 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074548 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671179 s, 15.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.472326 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104369 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101283 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220611 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263197 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26974 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196004 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285947 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239608 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363289 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354395 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307466 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373584 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301748 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294843 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394901 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377554 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723751 s, 145 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520349 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469853 s, 22.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666015 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249412 s, 42.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00625813 s, 168 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891456 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162042 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715363 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971919 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122054 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422144 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093543 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907927 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679442 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100304 s, 10.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (15s) == sanityn test 77d: check TRR nrs policy ================ 09:59:47 (1788875987) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.396812 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101113 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105516 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726399 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118395 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161104 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184979 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202758 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230086 s, 4.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273052 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240581 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264372 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258421 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225669 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275052 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31516 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34235 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381382 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103667 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00214241 s, 489 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712628 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07072 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722701 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08824 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535391 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112936 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453939 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802709 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168422 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873092 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326702 s, 32.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868703 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764599 s, 13.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.479166 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741678 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108256 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835497 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2507 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217645 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25785 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246452 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188739 s, 5.6 MB/sldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280236 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143348 s, 73.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296045 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872274 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11757 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15106 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100216 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541537 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14624 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473711 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175676 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157177 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00504688 s, 208 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390796 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429725 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145988 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378282 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150682 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34979 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394301 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400505 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00769484 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192872 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420004 s, 2.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.445398 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354269 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765823 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720245 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13079 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730968 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148686 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450646 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109998 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104045 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144845 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161982 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171546 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156234 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169554 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141598 s, 7.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266555 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286686 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277592 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246621 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365312 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391154 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372906 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397766 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07326 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373103 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453802 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459464 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505708 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509142 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567489 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589199 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629778 s, 16.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (14s) == sanityn test 77e: check TBF NID nrs policy ============ 10:00:01 (1788876001) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.59@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.471425 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102458 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107923 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907062 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121439 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147395 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140961 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108497 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105662 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218144 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248183 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237988 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325145 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307581 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232728 s, 4.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278227 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292376 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998646 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109246 s, 96.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353681 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134813 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774622 s, 13.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00235946 s, 444 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907063 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802944 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815403 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756186 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746729 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696934 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367069 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293951 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208781 s, 50.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.521126 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068687 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12301 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134062 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122361 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134018 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144744 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342777 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176086 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293564 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3421 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371529 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283976 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250921 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361971 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182876 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26076 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103466 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103099 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649793 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162859 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574064 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688454 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381863 s, 27.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380059 s, 27.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017338 s, 60.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558691 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661515 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539945 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327761 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316211 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00891537 s, 118 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790774 s, 13.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.465886 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894867 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108025 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901969 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634521 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272204 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215166 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218772 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11757 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169709 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134007 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235488 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289283 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308377 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282709 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303631 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279163 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00681575 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597056 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203927 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237042 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00956854 s, 110 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520267 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640702 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397153 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219151 s, 47.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103399 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371843 s, 282 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120173 s, 87.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00648279 s, 162 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123621 s, 84.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421239 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164143 s, 63.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.426697 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081045 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743935 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159751 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179696 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218081 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193888 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212527 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244609 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221034 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279549 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27618 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278303 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305117 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370577 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330973 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398847 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136554 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138136 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194507 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154113 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831928 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977993 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100954 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103177 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241445 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169906 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134844 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190136 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169405 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204209 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217896 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174905 s, 6.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (24s) == sanityn test 77f: check TBF JobID nrs policy ========== 10:00:25 (1788876025) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.574742 s, 29.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121107 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137834 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165953 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143962 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298757 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32771 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342597 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281174 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331999 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323592 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396279 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451387 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.47846 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39982 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429195 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.505525 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595092 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823679 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475954 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00884882 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259557 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058613 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155962 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598636 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603599 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625407 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637265 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828602 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142701 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349974 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926265 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828583 s, 12.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.641965 s, 26.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106277 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153124 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107533 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873241 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219891 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116935 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738679 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264843 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221898 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249566 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239179 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188161 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866821 s, 12.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23081 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235236 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255766 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270084 s, 38.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246734 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531457 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033111 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047588 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336935 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348763 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390651 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893952 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470467 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456337 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749936 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467411 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513667 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571809 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849671 s, 12.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.530251 s, 31.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703048 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107473 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12809 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188998 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215193 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283063 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272185 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340565 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32925 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278271 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382662 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308817 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313299 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375877 s, 2.8 MB/s execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348039 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426518 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577944 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050765 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544687 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627267 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511786 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689652 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551454 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563627 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674944 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665079 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101999 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439289 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997969 s, 10.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810776 s, 12.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847716 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951262 s, 11.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.503408 s, 33.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698015 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359122 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920592 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137759 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692627 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166265 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120319 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10811 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079875 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187787 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312649 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362523 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314364 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349769 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322031 s, 3.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328519 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039322 s, 26.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555411 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325401 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778072 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550985 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385242 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063469 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540343 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550242 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578557 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423344 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711962 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948375 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660523 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126229 s, 8.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103985 s, 10.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (24s) == sanityn test 77g: Change TBF type directly ============ 10:00:49 (1788876049) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 10:01:00 (1788876060) oleg359-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg359-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 oleg359-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg359-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg359-client: oleg359-server: ssh exited with exit code 22 oleg359-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg359-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg359-client: oleg359-server: ssh exited with exit code 22 oleg359-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg359-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg359-client: oleg359-server: ssh exited with exit code 22 PASS 77h (12s) == sanityn test 77i: Change rank of TBF rule ============= 10:01:12 (1788876072) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (22s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 10:01:34 (1788876094) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.749849 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133033 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100848 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23296 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310586 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319136 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348733 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38354 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327664 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398487 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397446 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426604 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384272 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43964 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378325 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457471 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494714 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376472 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380795 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309196 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393359 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302589 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306492 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311443 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339717 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399455 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421727 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331603 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412644 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427831 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345446 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378448 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436635 s, 2.4 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.22949 s, 11.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6466 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (64s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 10:02:38 (1788876158) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg359-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg359-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg359-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg359-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30533 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724235 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942631 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144645 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133034 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146585 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151492 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14325 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17347 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232337 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874796 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882504 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136598 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179202 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160792 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186769 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309222 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876381 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886114 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849825 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690739 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510632 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106023 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720139 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074361 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068289 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10917 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918763 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112083 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08378 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971272 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113074 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124603 s, 8.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9424 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9606 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg359-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg359-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg359-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg359-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30686 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147967 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146464 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156845 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224363 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200843 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205277 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261854 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252336 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266363 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265613 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301365 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298127 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342807 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356276 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355859 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400575 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569923 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694061 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436328 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816379 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832118 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793683 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851214 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798287 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801162 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721556 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608639 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799689 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691546 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604571 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872192 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100487 s, 10.4 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0702 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8108 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (154s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 10:05:13 (1788876313) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg359-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg359-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg359-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg359-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28425 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10965 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126353 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177812 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187423 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155273 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12553 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301963 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235021 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172548 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191821 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242896 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26124 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352529 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256121 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279868 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381736 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186636 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359649 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322707 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070036 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696662 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482059 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149678 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011996 s, 87.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187717 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107489 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104079 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116159 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084039 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112811 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920489 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145064 s, 7.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9575 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9667 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg359-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg359-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg359-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg359-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3018 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112265 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152563 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199067 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697483 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199651 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776432 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168442 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182167 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235419 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595798 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250642 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968201 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231282 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121308 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893511 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318924 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455786 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724198 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447958 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386008 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479784 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652418 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355498 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475619 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909403 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633922 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462392 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483565 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432394 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596286 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110451 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779304 s, 13.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9694 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9379 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (154s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 10:07:47 (1788876467) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.620385 s, 27.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11149 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108219 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102268 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125329 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224117 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266246 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237212 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274744 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212025 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32663 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296376 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326012 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365293 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350069 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384855 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32184 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349812 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071478 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190297 s, 55.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00921196 s, 114 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286431 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115755 s, 90.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163867 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025016 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708777 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102724 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129021 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740803 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795446 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143259 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142972 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020773 s, 50.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.85471 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8812 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.59@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.59@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.58343 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682617 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867995 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107772 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963593 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135515 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514055 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11072 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199315 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217291 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144671 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305829 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200133 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304268 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234863 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308136 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30199 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580211 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105094 s, 99.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00355757 s, 295 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495192 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849913 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941162 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00247407 s, 424 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834714 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105875 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315653 s, 33.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999495 s, 10.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133163 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114659 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178548 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107317 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126225 s, 8.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.06553 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97589 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.59@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.41244 s, 40.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106759 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984246 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182207 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228856 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23636 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247984 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237874 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26604 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316531 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255112 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332637 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274307 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284902 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306098 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336299 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361195 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115881 s, 90.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236887 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405847 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101852 s, 103 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955599 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0030953 s, 339 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00717311 s, 146 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267081 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701928 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717915 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00920502 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173872 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00561507 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06271 s, 16.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395163 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158161 s, 66.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.09787 s, 12.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09576 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.617067 s, 27.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446059 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634386 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919819 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742335 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966086 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948506 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478016 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148357 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174642 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154793 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169521 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13279 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402583 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390293 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354371 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4523 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540636 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789892 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735199 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235926 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481923 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437921 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338772 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149148 s, 70.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469941 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160377 s, 65.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626714 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620614 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626691 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751257 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370021 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190066 s, 55.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.868 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87663 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.36242 s, 12.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90998 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.428154 s, 39.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132019 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115384 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121137 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148181 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208149 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206288 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230632 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199485 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235403 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246001 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131669 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182766 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303207 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31456 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254568 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357196 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753144 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404307 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765879 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483858 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547568 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125296 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114579 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558343 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690934 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999414 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109011 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683098 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991398 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746166 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107622 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840884 s, 12.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9178 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6284 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.634848 s, 26.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619235 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845064 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112971 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122739 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159665 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205742 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220492 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196419 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264778 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319868 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315462 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290645 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376348 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307869 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25885 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427846 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394591 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650488 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04102 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276286 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713206 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345935 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384564 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359064 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587704 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666659 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561925 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607453 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601177 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598003 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700369 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785571 s, 13.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.02809 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91129 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.680677 s, 24.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933596 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155865 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177449 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212401 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185864 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264918 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303712 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256593 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344809 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348483 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385992 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326017 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360526 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430205 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.469752 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.517575 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415086 s, 25.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042183 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040165 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507361 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531794 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318734 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511199 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111791 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693241 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757288 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596905 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877599 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812624 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912987 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922455 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116192 s, 9.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9981 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0049 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.1162 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92864 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (438s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 10:15:06 (1788876906) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.52965 s, 31.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675564 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113175 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685788 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632977 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282509 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156036 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3008 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294442 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343483 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295387 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282291 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363487 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282321 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362128 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363073 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403984 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493389 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636327 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899374 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549989 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868651 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575267 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664576 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581272 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700802 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860774 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776988 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852296 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929904 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992063 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103302 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150157 s, 7.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.05041 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93275 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (55s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 10:16:00 (1788876960) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.59@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.59@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.772745 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102123 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740398 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892472 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152276 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237861 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243334 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268778 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219643 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344261 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36453 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404584 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409823 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406363 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440811 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343367 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375378 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130446 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185929 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164781 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13723 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146545 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156373 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154203 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175466 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164044 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20905 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185098 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209357 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17241 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164456 s, 6.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211304 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188626 s, 5.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg359-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg359-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6826 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9868 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (55s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 10:16:55 (1788877015) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.59@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.719039 s, 23.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116813 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132461 s, 7.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877721 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915387 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120538 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247856 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226697 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313814 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357397 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273223 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432638 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297819 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433208 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476933 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350914 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415259 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8e7d12dd3800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8e7d12dd3800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103793 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104976 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933769 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791788 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726917 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117139 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119448 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557566 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074292 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112946 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115952 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565706 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738742 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791644 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834366 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106142 s, 9.9 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg359-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg359-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6706 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17724 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (50s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 10:17:45 (1788877065) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg359-client.virtnet] [bs=1M] [count=16]